[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:34:45 dist-test-slave-8wc7 krb5kdc[2829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:34:45 dist-test-slave-8wc7 krb5kdc[2829](info): set up 2 sockets
Sep 24 10:34:45 dist-test-slave-8wc7 krb5kdc[2829](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 10:34:46.966168 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:34:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:34:47.093490 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36523
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:47.205829 2842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:47.206161 2842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:47.206238 2842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:47.210415 2842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:34:47.210518 2842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:47.210553 2842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:47.210595 2842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:34:47.210632 2842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:34:47.216096 2842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:36523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:47.217334 2842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:47.218472 2842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:47.227406 2848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:47.227401 2847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:47.227401 2850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:47.227517 2842 server_base.cc:1034] running on GCE node
I20260924 10:34:47.228230 2842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:47.229295 2842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:47.230453 2842 hybrid_clock.cc:648] HybridClock initialized: now 1790246087230417 us; error 55 us; skew 500 ppm
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:47.233107 2842 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:34:47.233987 2842 webserver.cc:460] Webserver started at http://127.2.193.190:42849/ using document root <none> and password file <none>
I20260924 10:34:47.234555 2842 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:47.234629 2842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:47.234874 2842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:47.236907 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "f041eba84142453cb83bacb48a2640ab"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "331789947e0ebc7fc49cef487fb13d75"
server_key_iv: "21b4ec94d8da5a342000a1e76e4ccdf9"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.237555 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "f041eba84142453cb83bacb48a2640ab"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "331789947e0ebc7fc49cef487fb13d75"
server_key_iv: "21b4ec94d8da5a342000a1e76e4ccdf9"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.241377 2842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:34:47.243741 2857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.244992 2842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:47.245154 2842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "f041eba84142453cb83bacb48a2640ab"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "331789947e0ebc7fc49cef487fb13d75"
server_key_iv: "21b4ec94d8da5a342000a1e76e4ccdf9"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.245322 2842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:47.267310 2842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:47.270242 2842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:47.270471 2842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:47.277869 2842 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:36523
I20260924 10:34:47.277884 2909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:36523 every 8 connection(s)
I20260924 10:34:47.278828 2842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:34:47.280010 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 2842
I20260924 10:34:47.280169 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:34:47.281810 2822 external_mini_cluster.cc:1428] Setting key 193da3be54249655eeb6c562559b175f
I20260924 10:34:47.282341 2910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:47.288923 2910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab: Bootstrap starting.
I20260924 10:34:47.291615 2910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:47.292356 2910 log.cc:824] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:47.294480 2910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab: No bootstrap required, opened a new log
I20260924 10:34:47.298487 2910 raft_consensus.cc:348] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f041eba84142453cb83bacb48a2640ab" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36523 } }
I20260924 10:34:47.298702 2910 raft_consensus.cc:374] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:47.298772 2910 raft_consensus.cc:729] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f041eba84142453cb83bacb48a2640ab, State: Initialized, Role: FOLLOWER
I20260924 10:34:47.299265 2910 consensus_queue.cc:260] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f041eba84142453cb83bacb48a2640ab" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36523 } }
I20260924 10:34:47.299388 2910 raft_consensus.cc:388] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:34:47.299471 2910 raft_consensus.cc:482] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:34:47.299577 2910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:47.300498 2910 raft_consensus.cc:504] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f041eba84142453cb83bacb48a2640ab" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36523 } }
I20260924 10:34:47.300892 2910 leader_election.cc:302] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f041eba84142453cb83bacb48a2640ab; no voters:
I20260924 10:34:47.301306 2910 leader_election.cc:288] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:34:47.301642 2915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:47.302139 2915 raft_consensus.cc:686] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [term 1 LEADER]: Becoming Leader. State: Replica: f041eba84142453cb83bacb48a2640ab, State: Running, Role: LEADER
I20260924 10:34:47.302469 2915 consensus_queue.cc:237] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f041eba84142453cb83bacb48a2640ab" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36523 } }
I20260924 10:34:47.302819 2910 sys_catalog.cc:564] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:34:47.304320 2916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f041eba84142453cb83bacb48a2640ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f041eba84142453cb83bacb48a2640ab" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36523 } } }
I20260924 10:34:47.304458 2916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:47.304754 2917 sys_catalog.cc:455] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f041eba84142453cb83bacb48a2640ab. Latest consensus state: current_term: 1 leader_uuid: "f041eba84142453cb83bacb48a2640ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f041eba84142453cb83bacb48a2640ab" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36523 } } }
I20260924 10:34:47.304847 2917 sys_catalog.cc:458] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:47.307415 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:47.285430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.2.193.190:36523)
0924 10:34:47.285910 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:47.285929 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:47.286952 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 10:34:47.290558 (+ 3606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:47.290578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:47.290956 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:47.291299 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:47.293488 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.294914 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.296041 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.296720 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.296898 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:47.300828 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:47.300835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:47.300852 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:47.300885 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:47.303158 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:47.303880 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:47.303886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:47.303892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:47.304046 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:47.304629 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:47.304633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:47.304634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:47.305213 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:47.305404 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:47.306292 (+ 888us) server_negotiation.cc:299] Negotiation successful
0924 10:34:47.306535 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":132,"threads_started":1}
W20260924 10:34:47.308647 2930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:34:47.308734 2930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:34:47.308799 2927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:34:47.309551 2927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:34:47.315750 2927 catalog_manager.cc:1347] Generated new cluster ID: 8101384a0d414249aee6e802e3aa512d
I20260924 10:34:47.315814 2927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:34:47.328547 2927 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:34:47.329576 2927 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:34:47.338572 2927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f041eba84142453cb83bacb48a2640ab: Generated new TSK 0
I20260924 10:34:47.339308 2927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:34:47.374392 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36523
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:47.488265 2936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:47.488521 2936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:47.488565 2936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:47.492686 2936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:47.492758 2936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:47.492879 2936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:34:47.498109 2936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:47.499130 2936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:47.500200 2936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:47.510007 2942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:47.510007 2941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:47.510136 2936 server_base.cc:1034] running on GCE node
W20260924 10:34:47.510363 2944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:47.510779 2936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:47.511415 2936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:47.512578 2936 hybrid_clock.cc:648] HybridClock initialized: now 1790246087512545 us; error 47 us; skew 500 ppm
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:47.515282 2936 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:34:47.516126 2936 webserver.cc:460] Webserver started at http://127.2.193.129:43703/ using document root <none> and password file <none>
I20260924 10:34:47.516685 2936 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:47.516774 2936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:47.517005 2936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:47.519073 2936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "54dfb69b84774819a6e6bc2585505eda"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "8ed1efcf938aed7ae690dde97d018e78"
server_key_iv: "09a07b24dc34cc1ef2e2ca9767ab8c06"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.519619 2936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "54dfb69b84774819a6e6bc2585505eda"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "8ed1efcf938aed7ae690dde97d018e78"
server_key_iv: "09a07b24dc34cc1ef2e2ca9767ab8c06"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.523481 2936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 10:34:47.525756 2951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.526896 2936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:34:47.527029 2936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "54dfb69b84774819a6e6bc2585505eda"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "8ed1efcf938aed7ae690dde97d018e78"
server_key_iv: "09a07b24dc34cc1ef2e2ca9767ab8c06"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.527133 2936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:47.546603 2936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:47.549475 2936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:47.549665 2936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:47.550235 2936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:34:47.551127 2936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:47.551180 2936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.551227 2936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:47.551250 2936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.559922 2936 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:39827
I20260924 10:34:47.559952 3064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:39827 every 8 connection(s)
I20260924 10:34:47.560791 2936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:47.570941 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 2936
I20260924 10:34:47.571066 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:34:47.571389 2822 external_mini_cluster.cc:1428] Setting key a4fbc5e5b9a0c750ccbaf7c3572ba452
I20260924 10:34:47.573137 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:47.562496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:41331 (local address 127.2.193.190:36523)
0924 10:34:47.562620 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:47.562623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:47.563366 (+ 743us) server_negotiation.cc:407] Connection header received
0924 10:34:47.564607 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:47.564609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:47.564656 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:47.564726 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:47.565816 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.566332 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.567395 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.567613 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.567687 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:47.569832 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:47.569835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:47.569837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:47.569859 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:47.571101 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:47.571652 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:47.571654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:47.571655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:47.571727 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:47.572084 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:47.572088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:47.572089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:47.572278 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:47.572384 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:47.572854 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 10:34:47.572978 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:34:47.573963 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:47.562815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36523 (local address 127.2.193.129:41331)
0924 10:34:47.563215 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:47.563253 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:47.564410 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:47.564851 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:47.564857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:47.565223 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:47.565690 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:47.565703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.566430 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.566432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:47.567283 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:47.567291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.567727 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.567729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:47.567855 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:47.568321 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:47.568335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:47.569692 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:47.571216 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:47.571222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:47.571235 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:47.571560 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:47.571819 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:47.571822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:47.571824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:47.571980 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:47.572387 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:47.572397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:47.572632 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 10:34:47.572815 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 10:34:47.573128 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260924 10:34:47.575220 3065 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36523
I20260924 10:34:47.575455 3065 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:47.575949 3065 heartbeater.cc:511] Master 127.2.193.190:36523 requested a full tablet report, sending...
I20260924 10:34:47.577522 2874 ts_manager.cc:194] Registered new tserver with Master: 54dfb69b84774819a6e6bc2585505eda (127.2.193.129:39827)
I20260924 10:34:47.578632 2874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:41331
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:34:47.597815 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36523
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:47.706103 3070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:47.706350 3070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:47.706394 3070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:47.710354 3070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:47.710408 3070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:47.710525 3070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:34:47.715567 3070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:47.716604 3070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:47.717612 3070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:47.726703 3078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:47.726712 3075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:47.726749 3076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:47.727069 3070 server_base.cc:1034] running on GCE node
I20260924 10:34:47.727515 3070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:47.728073 3070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:47.729249 3070 hybrid_clock.cc:648] HybridClock initialized: now 1790246087729232 us; error 26 us; skew 500 ppm
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:47.731958 3070 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:34:47.732802 3070 webserver.cc:460] Webserver started at http://127.2.193.130:34447/ using document root <none> and password file <none>
I20260924 10:34:47.733404 3070 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:47.733471 3070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:47.733716 3070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:47.736073 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "94a1e809d04243e68123cdec91215932"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "0837fa72eee481420be3bc75ddbd7391"
server_key_iv: "f5e3aacdf5c549e635975710cbf56fdb"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.736582 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "94a1e809d04243e68123cdec91215932"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "0837fa72eee481420be3bc75ddbd7391"
server_key_iv: "f5e3aacdf5c549e635975710cbf56fdb"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.740566 3070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:34:47.742996 3085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.744252 3070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:34:47.744375 3070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "94a1e809d04243e68123cdec91215932"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "0837fa72eee481420be3bc75ddbd7391"
server_key_iv: "f5e3aacdf5c549e635975710cbf56fdb"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.744499 3070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:47.758400 3070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:47.761251 3070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:47.761442 3070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:47.762008 3070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:34:47.762811 3070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:47.762881 3070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.762949 3070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:47.762995 3070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.773169 3070 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:38275
I20260924 10:34:47.773224 3198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:38275 every 8 connection(s)
I20260924 10:34:47.774084 3070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:47.783823 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 3070
I20260924 10:34:47.783963 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:34:47.784286 2822 external_mini_cluster.cc:1428] Setting key 221dd058c4ceab6821c9965ff79759bb
I20260924 10:34:47.788386 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:47.775802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:33289 (local address 127.2.193.190:36523)
0924 10:34:47.775938 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:47.775943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:47.776758 (+ 815us) server_negotiation.cc:407] Connection header received
0924 10:34:47.778118 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:47.778122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:47.778180 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:47.778286 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:47.779414 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.779950 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.781099 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.781390 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.781516 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:47.784008 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:47.784015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:47.784021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:47.784048 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:47.785976 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:47.786606 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:47.786608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:47.786610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:47.786722 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:47.787105 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:47.787107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:47.787108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:47.787321 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:47.787397 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:47.788019 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 10:34:47.788177 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:34:47.789280 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:47.776183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36523 (local address 127.2.193.130:33289)
0924 10:34:47.776609 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:47.776650 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:47.777896 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:47.778459 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:47.778467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:47.778801 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:47.779264 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:47.779276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.780074 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.780079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:47.780973 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:47.780981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:47.781552 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:47.781556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:47.781722 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:47.782318 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:47.782333 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:47.783802 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:47.786118 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:47.786124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:47.786138 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:47.786495 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:47.786827 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:47.786830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:47.786831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:47.787012 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:47.787421 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:47.787431 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:47.787765 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 10:34:47.787982 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 10:34:47.788348 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260924 10:34:47.790485 3199 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36523
I20260924 10:34:47.790732 3199 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:47.791213 3199 heartbeater.cc:511] Master 127.2.193.190:36523 requested a full tablet report, sending...
I20260924 10:34:47.792291 2874 ts_manager.cc:194] Registered new tserver with Master: 94a1e809d04243e68123cdec91215932 (127.2.193.130:38275)
I20260924 10:34:47.793108 2874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:33289
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:34:47.812319 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36523
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:47.926133 3204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:47.926385 3204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:47.926429 3204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:47.930572 3204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:47.930631 3204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:47.930708 3204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:34:47.936110 3204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:47.937225 3204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:47.938241 3204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:47.948097 3212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:47.948117 3210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:47.948071 3209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:47.948482 3204 server_base.cc:1034] running on GCE node
I20260924 10:34:47.948915 3204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:47.949622 3204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:47.950775 3204 hybrid_clock.cc:648] HybridClock initialized: now 1790246087950772 us; error 57 us; skew 500 ppm
Sep 24 10:34:47 dist-test-slave-8wc7 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:47.953485 3204 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:34:47.954325 3204 webserver.cc:460] Webserver started at http://127.2.193.131:41825/ using document root <none> and password file <none>
I20260924 10:34:47.954861 3204 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:47.954919 3204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:47.955153 3204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:47.957118 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "d2ab12ab19f3415583a74a0bbb37b1aa"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "fc245823f19fc18b72c44c32d8734dcd"
server_key_iv: "584d6a3882fdb9e423ffa908f1b3c857"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.957746 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "d2ab12ab19f3415583a74a0bbb37b1aa"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "fc245823f19fc18b72c44c32d8734dcd"
server_key_iv: "584d6a3882fdb9e423ffa908f1b3c857"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.961519 3204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:34:47.963907 3219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:47.965101 3204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:47.965267 3204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "d2ab12ab19f3415583a74a0bbb37b1aa"
format_stamp: "Formatted at 2026-09-24 10:34:47 on dist-test-slave-8wc7"
server_key: "fc245823f19fc18b72c44c32d8734dcd"
server_key_iv: "584d6a3882fdb9e423ffa908f1b3c857"
server_key_version: "encryptionkey@0"
I20260924 10:34:47.965387 3204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:47.997391 3204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:48.000360 3204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:48.000561 3204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:48.001140 3204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:34:48.001998 3204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:48.002066 3204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:48.002133 3204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:48.002174 3204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:48.011935 3204 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:33821
I20260924 10:34:48.011952 3332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:33821 every 8 connection(s)
I20260924 10:34:48.012861 3204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:34:48.018512 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 3204
I20260924 10:34:48.018658 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:34:48.019002 2822 external_mini_cluster.cc:1428] Setting key d60e7209dbb5eba158ee6618f25967e7
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:48.026381 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:48.014664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:33181 (local address 127.2.193.190:36523)
0924 10:34:48.014812 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:48.014815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:48.016073 (+ 1258us) server_negotiation.cc:407] Connection header received
0924 10:34:48.016895 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:48.016898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:48.016944 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:48.017025 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:48.018164 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.018740 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.020350 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.020565 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.020670 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:48.023128 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:48.023132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:48.023135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:48.023164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:48.024395 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:48.024899 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:48.024904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:48.024906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:48.024996 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:48.025412 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:48.025416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:48.025418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:48.025621 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:48.025704 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:48.026161 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 10:34:48.026259 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:34:48.027168 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:48.015047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36523 (local address 127.2.193.131:33181)
0924 10:34:48.015913 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:48.015955 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:48.016711 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:48.017207 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:48.017214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:48.017566 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:48.018037 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:48.018049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.018882 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.018887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:48.020237 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:48.020248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.020711 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.020719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:48.020870 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:48.021498 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:48.021514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:48.022974 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:48.024513 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:48.024518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:48.024533 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:48.024800 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:48.025088 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:48.025092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:48.025094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:48.025293 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:48.025714 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:48.025719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:48.025944 (+ 225us) client_negotiation.cc:769] Sending connection context
0924 10:34:48.026101 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 10:34:48.026345 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":538,"threads_started":1}
I20260924 10:34:48.028316 3333 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36523
I20260924 10:34:48.028558 3333 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:48.029058 3333 heartbeater.cc:511] Master 127.2.193.190:36523 requested a full tablet report, sending...
I20260924 10:34:48.030151 2874 ts_manager.cc:194] Registered new tserver with Master: d2ab12ab19f3415583a74a0bbb37b1aa (127.2.193.131:33821)
I20260924 10:34:48.030787 2874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:33181
I20260924 10:34:48.033367 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246088, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246088, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:34:48.059744 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:48.050740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.2.193.129:39827)
0924 10:34:48.051087 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:48.051100 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:48.051126 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:34:48.051254 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:48.051259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:48.051463 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:48.051587 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:48.052206 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.053002 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.053847 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.054239 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.054332 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:48.056591 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:48.056600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:48.056614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:48.056637 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:48.058165 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:48.058544 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:48.058550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:48.058551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:48.058624 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:48.058836 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:48.058838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:48.058839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:48.059058 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:48.059228 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:48.059389 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 10:34:48.059559 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":138,"threads_started":1}
W20260924 10:34:48.060703 2979 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35410
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:34:48.087596 3340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:48.078256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.2.193.129:39827)
0924 10:34:48.078425 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:48.078430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:48.078564 (+ 134us) server_negotiation.cc:407] Connection header received
0924 10:34:48.078736 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:48.078740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:48.078792 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:48.078886 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:48.079593 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.080113 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.080946 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:48.081245 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:48.081353 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:48.084421 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:48.084425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:48.084427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:48.084464 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:48.086160 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:48.086563 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:48.086566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:48.086567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:48.086642 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:48.086860 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:48.086862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:48.086863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:48.087017 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:48.087111 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:48.087295 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:34:48.087397 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:34:48.088680 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 2936
I20260924 10:34:48.094966 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 3070
I20260924 10:34:48.101940 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 3204
I20260924 10:34:48.107115 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 2842
2026-09-24T10:34:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2205 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[3347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[3347](info): set up 2 sockets
Sep 24 10:34:48 dist-test-slave-8wc7 krb5kdc[3347](info): commencing operation
krb5kdc: starting...
W20260924 10:34:49.132848 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:34:49 dist-test-slave-8wc7 krb5kdc[3347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:34:49.248466 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:43987
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:43987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:49.355906 3360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:49.356181 3360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:49.356247 3360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:49.360183 3360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:34:49.360258 3360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:49.360296 3360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:49.360350 3360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:34:49.360386 3360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:34:49.365463 3360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:43987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:43987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:49.366474 3360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:49.367456 3360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:49.375445 3366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:49.375533 3365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:49.375485 3368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:49.375665 3360 server_base.cc:1034] running on GCE node
I20260924 10:34:49.376221 3360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:49.377367 3360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:49.378545 3360 hybrid_clock.cc:648] HybridClock initialized: now 1790246089378517 us; error 41 us; skew 500 ppm
Sep 24 10:34:49 dist-test-slave-8wc7 krb5kdc[3347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:49.381011 3360 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:34:49.381848 3360 webserver.cc:460] Webserver started at http://127.2.193.190:42947/ using document root <none> and password file <none>
I20260924 10:34:49.382372 3360 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:49.382471 3360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:49.382690 3360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:49.384663 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "4478169025454e04aa135877903a7d30"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "51ad761f87832c353edfaa24c4bd88f9"
server_key_iv: "c6e5dae050364a52400c1877cadd3ceb"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.385267 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "4478169025454e04aa135877903a7d30"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "51ad761f87832c353edfaa24c4bd88f9"
server_key_iv: "c6e5dae050364a52400c1877cadd3ceb"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.388669 3360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:34:49.390996 3375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:49.392093 3360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:49.392226 3360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "4478169025454e04aa135877903a7d30"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "51ad761f87832c353edfaa24c4bd88f9"
server_key_iv: "c6e5dae050364a52400c1877cadd3ceb"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.392328 3360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:49.401314 3360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:49.403699 3360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:49.403898 3360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:49.410353 3360 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:43987
I20260924 10:34:49.410378 3427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:43987 every 8 connection(s)
I20260924 10:34:49.411204 3360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:34:49.413633 3428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:49.414095 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 3360
I20260924 10:34:49.414177 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:34:49.414438 2822 external_mini_cluster.cc:1428] Setting key 7b875c35ada9061f14f5800eee97a2d3
I20260924 10:34:49.420333 3428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30: Bootstrap starting.
I20260924 10:34:49.422627 3428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30: Neither blocks nor log segments found. Creating new log.
Sep 24 10:34:49 dist-test-slave-8wc7 krb5kdc[3347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:49.423509 3428 log.cc:824] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:49.425565 3428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30: No bootstrap required, opened a new log
I20260924 10:34:49.428300 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:49.415626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.2.193.190:43987)
0924 10:34:49.416015 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:49.416033 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:49.416073 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:34:49.416832 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:49.416856 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:49.417267 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:49.417616 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:49.418241 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.419702 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.420637 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.421303 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.421486 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:49.424004 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:49.424010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:49.424025 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:49.424057 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:49.425802 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:49.426250 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:49.426255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:49.426259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:49.426354 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:49.426638 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:49.426640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:49.426641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:49.426966 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:49.427096 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:49.427382 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:34:49.427642 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260924 10:34:49.429430 3428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4478169025454e04aa135877903a7d30" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43987 } }
I20260924 10:34:49.429668 3428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:49.429724 3428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4478169025454e04aa135877903a7d30, State: Initialized, Role: FOLLOWER
I20260924 10:34:49.430147 3428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4478169025454e04aa135877903a7d30" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43987 } }
I20260924 10:34:49.430321 3428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:34:49.430401 3428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:34:49.430518 3428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:49.431308 3428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4478169025454e04aa135877903a7d30" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43987 } }
I20260924 10:34:49.431628 3428 leader_election.cc:302] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4478169025454e04aa135877903a7d30; no voters:
I20260924 10:34:49.431901 3428 leader_election.cc:288] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:34:49.432004 3433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:49.432216 3433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [term 1 LEADER]: Becoming Leader. State: Replica: 4478169025454e04aa135877903a7d30, State: Running, Role: LEADER
I20260924 10:34:49.432528 3433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4478169025454e04aa135877903a7d30" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43987 } }
I20260924 10:34:49.433094 3428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:34:49.434336 3435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4478169025454e04aa135877903a7d30. Latest consensus state: current_term: 1 leader_uuid: "4478169025454e04aa135877903a7d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4478169025454e04aa135877903a7d30" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43987 } } }
I20260924 10:34:49.434332 3434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4478169025454e04aa135877903a7d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4478169025454e04aa135877903a7d30" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43987 } } }
I20260924 10:34:49.434492 3435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:49.434492 3434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30 [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:49.435257 3443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:34:49.437844 3443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:34:49.442883 3443 catalog_manager.cc:1347] Generated new cluster ID: 70bc99abddad433ea955991980593b08
I20260924 10:34:49.442941 3443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:34:49.452253 3443 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:34:49.453100 3443 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:34:49.459383 3443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4478169025454e04aa135877903a7d30: Generated new TSK 0
I20260924 10:34:49.460078 3443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:34:49.493978 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:43987
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:49.602067 3454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:49.602311 3454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:49.602355 3454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:49.606405 3454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:49.606465 3454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:49.606554 3454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:34:49.611773 3454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:43987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:49.612815 3454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:49.613862 3454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:49.623236 3459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:49.623284 3462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:49.623242 3460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:49.623566 3454 server_base.cc:1034] running on GCE node
I20260924 10:34:49.623939 3454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:49.624540 3454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:49.625691 3454 hybrid_clock.cc:648] HybridClock initialized: now 1790246089625675 us; error 26 us; skew 500 ppm
Sep 24 10:34:49 dist-test-slave-8wc7 krb5kdc[3347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:49.628391 3454 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:34:49.629221 3454 webserver.cc:460] Webserver started at http://127.2.193.129:45641/ using document root <none> and password file <none>
I20260924 10:34:49.629722 3454 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:49.629776 3454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:49.629927 3454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:49.631796 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "23534aa55cec40a086b9348d5691bd29"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "8d6867ff05bd8f7da0e96e63695b0b2a"
server_key_iv: "355b9836fb4966c3bdc71115526a1380"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.632273 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "23534aa55cec40a086b9348d5691bd29"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "8d6867ff05bd8f7da0e96e63695b0b2a"
server_key_iv: "355b9836fb4966c3bdc71115526a1380"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.636148 3454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 10:34:49.638339 3469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:49.639447 3454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:34:49.639575 3454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "23534aa55cec40a086b9348d5691bd29"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "8d6867ff05bd8f7da0e96e63695b0b2a"
server_key_iv: "355b9836fb4966c3bdc71115526a1380"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.639678 3454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:49.651348 3454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:49.654491 3454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:49.654683 3454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:49.655339 3454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:34:49.656212 3454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:49.656257 3454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:49.656296 3454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:49.656318 3454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:49.665020 3454 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38509
I20260924 10:34:49.665040 3582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38509 every 8 connection(s)
I20260924 10:34:49.665884 3454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:34:49.669644 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 3454
I20260924 10:34:49.669765 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:34:49.670028 2822 external_mini_cluster.cc:1428] Setting key a7424dd52f97a5578ac3444943712100
Sep 24 10:34:49 dist-test-slave-8wc7 krb5kdc[3347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:34:49.680836 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:49.667710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:48713 (local address 127.2.193.190:43987)
0924 10:34:49.667852 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:49.667856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:49.668553 (+ 697us) server_negotiation.cc:407] Connection header received
0924 10:34:49.670001 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:49.670004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:49.670061 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:49.670169 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:49.671538 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.672112 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.673676 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.674081 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.674215 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:49.677121 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:49.677124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:49.677126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:49.677149 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:49.678649 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:49.679277 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:49.679280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:49.679281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:49.679349 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:49.679759 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:49.679761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:49.679762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:49.679935 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:49.680016 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:49.680593 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 10:34:49.680712 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:34:49.681795 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:49.667966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:43987 (local address 127.2.193.129:48713)
0924 10:34:49.668390 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:49.668432 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:49.669717 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:49.670363 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:49.670372 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:49.670783 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:49.671372 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:49.671387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.672241 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.672247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:49.673555 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:49.673564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.674228 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.674231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:49.674361 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:49.674960 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:49.674977 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:49.676934 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:49.678777 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:49.678787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:49.678807 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:49.679172 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:49.679452 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:49.679456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:49.679460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:49.679659 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:49.680034 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:49.680039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:49.680285 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 10:34:49.680529 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 10:34:49.680801 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:34:49.683094 3583 heartbeater.cc:347] Connected to a master server at 127.2.193.190:43987
I20260924 10:34:49.683349 3583 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:49.683859 3583 heartbeater.cc:511] Master 127.2.193.190:43987 requested a full tablet report, sending...
I20260924 10:34:49.685415 3392 ts_manager.cc:194] Registered new tserver with Master: 23534aa55cec40a086b9348d5691bd29 (127.2.193.129:38509)
I20260924 10:34:49.686545 3392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:48713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:34:49.697757 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:43987
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:49.805577 3588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:49.805843 3588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:49.805887 3588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:49.809834 3588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:49.809894 3588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:49.809954 3588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:34:49.815212 3588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:43987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:49.816206 3588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:49.817196 3588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:49.826360 3596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:49.826324 3594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:49.826347 3588 server_base.cc:1034] running on GCE node
W20260924 10:34:49.826367 3593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:49.826930 3588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:49.827557 3588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:49.828742 3588 hybrid_clock.cc:648] HybridClock initialized: now 1790246089828732 us; error 31 us; skew 500 ppm
Sep 24 10:34:49 dist-test-slave-8wc7 krb5kdc[3347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:49.831485 3588 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:34:49.832322 3588 webserver.cc:460] Webserver started at http://127.2.193.130:36071/ using document root <none> and password file <none>
I20260924 10:34:49.832872 3588 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:49.833003 3588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:49.833292 3588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:49.835480 3588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "3ad2bf58c95b4a7c8323aaa6f396d903"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "9a7b3941bba49a9575310e50e6fc4d85"
server_key_iv: "aeef04e75118cdb0ad25575d60be2ff1"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.836046 3588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "3ad2bf58c95b4a7c8323aaa6f396d903"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "9a7b3941bba49a9575310e50e6fc4d85"
server_key_iv: "aeef04e75118cdb0ad25575d60be2ff1"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.839690 3588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:34:49.842161 3603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:49.843379 3588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:34:49.843503 3588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "3ad2bf58c95b4a7c8323aaa6f396d903"
format_stamp: "Formatted at 2026-09-24 10:34:49 on dist-test-slave-8wc7"
server_key: "9a7b3941bba49a9575310e50e6fc4d85"
server_key_iv: "aeef04e75118cdb0ad25575d60be2ff1"
server_key_version: "encryptionkey@0"
I20260924 10:34:49.843611 3588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:49.858410 3588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:49.861696 3588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:49.861894 3588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:49.862500 3588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:34:49.863288 3588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:49.863356 3588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:49.863425 3588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:49.863492 3588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:49.871861 3588 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:46307
I20260924 10:34:49.871887 3716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:46307 every 8 connection(s)
I20260924 10:34:49.872721 3588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:34:49.873812 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 3588
I20260924 10:34:49.873920 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:34:49.874207 2822 external_mini_cluster.cc:1428] Setting key b051136b918eb0bf5f1b247accd667af
Sep 24 10:34:49 dist-test-slave-8wc7 krb5kdc[3347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
I20260924 10:34:49.886533 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:49.874589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:54471 (local address 127.2.193.190:43987)
0924 10:34:49.874734 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:49.874740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:49.875633 (+ 893us) server_negotiation.cc:407] Connection header received
0924 10:34:49.876995 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:49.876999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:49.877049 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:49.877130 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:49.878294 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.878841 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.879966 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.880193 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.880301 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:49.882736 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:49.882739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:49.882741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:49.882764 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:49.884154 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:49.884659 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:49.884664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:49.884666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:49.884744 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:49.885089 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:49.885094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:49.885096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:49.885514 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:49.885623 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:49.886131 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 10:34:49.886271 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:34:49.887138 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:49.875040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:43987 (local address 127.2.193.130:54471)
0924 10:34:49.875493 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:49.875543 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:49.876776 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:49.877335 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:49.877342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:49.877691 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:49.878156 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:49.878168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.878964 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.878968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:49.879838 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:49.879847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:49.880337 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:49.880341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:49.880513 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:49.881102 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:49.881118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:49.882539 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:49.884285 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:49.884291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:49.884306 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:49.884557 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:49.884841 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:49.884844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:49.884847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:49.884995 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:49.885627 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:49.885632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:49.885864 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 10:34:49.886052 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 10:34:49.886291 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260924 10:34:49.888309 3717 heartbeater.cc:347] Connected to a master server at 127.2.193.190:43987
I20260924 10:34:49.888547 3717 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:49.889019 3717 heartbeater.cc:511] Master 127.2.193.190:43987 requested a full tablet report, sending...
I20260924 10:34:49.890132 3392 ts_manager.cc:194] Registered new tserver with Master: 3ad2bf58c95b4a7c8323aaa6f396d903 (127.2.193.130:46307)
I20260924 10:34:49.891053 3392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:54471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:34:49.900303 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:43987
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:50.011178 3722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:50.011458 3722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:50.011504 3722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:50.015542 3722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:50.015606 3722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:50.015669 3722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:34:50.021064 3722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:43987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:50.022208 3722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:50.023392 3722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:50.032806 3728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:50.032776 3727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:50.032796 3730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:50.033003 3722 server_base.cc:1034] running on GCE node
I20260924 10:34:50.033474 3722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:50.034204 3722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:50.035382 3722 hybrid_clock.cc:648] HybridClock initialized: now 1790246090035363 us; error 27 us; skew 500 ppm
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:50.038228 3722 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:34:50.039052 3722 webserver.cc:460] Webserver started at http://127.2.193.131:42143/ using document root <none> and password file <none>
I20260924 10:34:50.039601 3722 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:50.039659 3722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:50.039891 3722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:50.041976 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "a16ab99afb2443b3b8475f1307036d59"
format_stamp: "Formatted at 2026-09-24 10:34:50 on dist-test-slave-8wc7"
server_key: "f618f1cfc20b08a5b823d7020186e4d8"
server_key_iv: "a2e7f54cc10579e448e239d8fdb6f63b"
server_key_version: "encryptionkey@0"
I20260924 10:34:50.042531 3722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "a16ab99afb2443b3b8475f1307036d59"
format_stamp: "Formatted at 2026-09-24 10:34:50 on dist-test-slave-8wc7"
server_key: "f618f1cfc20b08a5b823d7020186e4d8"
server_key_iv: "a2e7f54cc10579e448e239d8fdb6f63b"
server_key_version: "encryptionkey@0"
I20260924 10:34:50.046252 3722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 10:34:50.048641 3737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:50.049811 3722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:34:50.049939 3722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "a16ab99afb2443b3b8475f1307036d59"
format_stamp: "Formatted at 2026-09-24 10:34:50 on dist-test-slave-8wc7"
server_key: "f618f1cfc20b08a5b823d7020186e4d8"
server_key_iv: "a2e7f54cc10579e448e239d8fdb6f63b"
server_key_version: "encryptionkey@0"
I20260924 10:34:50.050057 3722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:50.060822 3722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:50.064112 3722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:50.064312 3722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:50.064877 3722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:34:50.065734 3722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:50.065804 3722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:50.065873 3722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:50.065917 3722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:50.074254 3722 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:45063
I20260924 10:34:50.074277 3850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:45063 every 8 connection(s)
I20260924 10:34:50.075112 3722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:34:50.077037 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 3722
I20260924 10:34:50.077152 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:34:50.077430 2822 external_mini_cluster.cc:1428] Setting key dc32dbe5e821228f9209fd282baccef2
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:50.088416 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.076885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:39741 (local address 127.2.193.190:43987)
0924 10:34:50.077065 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.077071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.078041 (+ 970us) server_negotiation.cc:407] Connection header received
0924 10:34:50.079668 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.079672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.079729 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.079790 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:50.080937 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.081496 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.082573 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.082775 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.082859 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.085251 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:50.085254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:50.085256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:50.085278 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:50.086493 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.087148 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.087150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.087151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.087221 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.087532 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.087534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.087535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.087695 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:50.087768 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.088196 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.088295 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:34:50.089308 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.077358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:43987 (local address 127.2.193.131:39741)
0924 10:34:50.077880 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:50.077922 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:50.079436 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:50.079954 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:50.079961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:50.080317 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:50.080794 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.080808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.081598 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.081601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.082465 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.082476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.082895 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.082897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.083041 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.083647 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:50.083663 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:50.085058 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:50.086589 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:50.086594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:50.086606 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:50.087053 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:50.087305 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:50.087307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:50.087308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:50.087452 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:50.087778 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:50.087782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:50.088002 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 10:34:50.088155 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 10:34:50.088393 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":133,"threads_started":1}
I20260924 10:34:50.090771 3851 heartbeater.cc:347] Connected to a master server at 127.2.193.190:43987
I20260924 10:34:50.091040 3851 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:50.091609 3851 heartbeater.cc:511] Master 127.2.193.190:43987 requested a full tablet report, sending...
I20260924 10:34:50.092736 3392 ts_manager.cc:194] Registered new tserver with Master: a16ab99afb2443b3b8475f1307036d59 (127.2.193.131:45063)
I20260924 10:34:50.093386 3392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:39741
I20260924 10:34:50.103551 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:34:50.114687 3431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.107027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.2.193.190:43987)
0924 10:34:50.107187 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.107192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.107365 (+ 173us) server_negotiation.cc:407] Connection header received
0924 10:34:50.107517 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.107521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.107570 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.107656 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:50.108334 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.108853 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.109738 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.109955 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.110066 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.111465 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:50.111476 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:50.111479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:50.111510 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:50.112915 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.113421 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.113429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.113430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.113511 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.113796 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.113799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.113800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.114022 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:50.114165 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.114395 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.114521 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:34:50.119671 3392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:34:50.122771 3392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:50.137080 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.130998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.2.193.131:45063)
0924 10:34:50.131703 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.131713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.131737 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:34:50.131820 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.131826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.132052 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.132204 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:50.132928 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.133974 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.135852 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.136394 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.136534 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.136681 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.136826 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.136917 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":111,"threads_started":1}
I20260924 10:34:50.137079 3863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.130878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:45063 (local address 127.0.0.1:37414)
0924 10:34:50.131480 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:50.131528 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:50.131642 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:50.132390 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:50.132395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:50.132434 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:50.132820 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.132828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.134119 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.134122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.135715 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.135726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.136534 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.136537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.136641 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.136676 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:34:50.136787 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:34:50.136918 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":224,"spinlock_wait_cycles":8320,"thread_start_us":88,"threads_started":1}
I20260924 10:34:50.137125 3862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.130951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38509 (local address 127.0.0.1:40184)
0924 10:34:50.131731 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:50.131745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:50.131839 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:50.132390 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:50.132395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:50.132429 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:50.132596 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.132604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.134080 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.134084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.135715 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.135726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.136702 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.136706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.136791 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.136812 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:34:50.136863 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 10:34:50.136941 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":99,"spinlock_wait_cycles":2816,"thread_start_us":100,"threads_started":1}
I20260924 10:34:50.137619 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.131585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46307 (local address 127.0.0.1:44708)
0924 10:34:50.132609 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:50.132621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:50.132699 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:50.133137 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:50.133140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:50.133158 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:50.133258 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.133264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.134463 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.134467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.135633 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.135642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.136867 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.136871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.136946 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.136964 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:34:50.137030 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:34:50.137083 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":22,"thread_start_us":103,"threads_started":1}
I20260924 10:34:50.138047 3867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.132086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.2.193.130:46307)
0924 10:34:50.132713 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.132724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.132744 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:34:50.132833 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.132838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.132999 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.133142 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:50.133368 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.134348 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.135746 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.136566 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.136680 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.136811 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.137330 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.137538 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":106,"threads_started":1}
I20260924 10:34:50.138247 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.131326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.2.193.129:38509)
0924 10:34:50.131592 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.131605 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.131829 (+ 224us) server_negotiation.cc:407] Connection header received
0924 10:34:50.131947 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.131952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.132140 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.132289 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:50.132723 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.133938 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.135813 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.136566 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.137611 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.137920 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.137980 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.138073 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260924 10:34:50.140293 3785 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee4d9ab0258418a89a2a724f6b028a5 (DEFAULT_TABLE table=test-table [id=071b3942fd24431aa40c7885b79ae8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:50.140702 3517 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee4d9ab0258418a89a2a724f6b028a5 (DEFAULT_TABLE table=test-table [id=071b3942fd24431aa40c7885b79ae8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:50.141502 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee4d9ab0258418a89a2a724f6b028a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:50.141755 3517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee4d9ab0258418a89a2a724f6b028a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:50.142555 3651 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee4d9ab0258418a89a2a724f6b028a5 (DEFAULT_TABLE table=test-table [id=071b3942fd24431aa40c7885b79ae8a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:50.143441 3651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ee4d9ab0258418a89a2a724f6b028a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:50.148996 3868 tablet_bootstrap.cc:492] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29: Bootstrap starting.
I20260924 10:34:50.149807 3869 tablet_bootstrap.cc:492] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59: Bootstrap starting.
I20260924 10:34:50.152186 3868 tablet_bootstrap.cc:654] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:50.152648 3870 tablet_bootstrap.cc:492] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903: Bootstrap starting.
I20260924 10:34:50.152678 3869 tablet_bootstrap.cc:654] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:50.153618 3868 log.cc:824] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:50.153618 3869 log.cc:824] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:50.155433 3870 tablet_bootstrap.cc:654] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:50.155869 3868 tablet_bootstrap.cc:492] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29: No bootstrap required, opened a new log
I20260924 10:34:50.155880 3869 tablet_bootstrap.cc:492] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59: No bootstrap required, opened a new log
I20260924 10:34:50.156090 3868 ts_tablet_manager.cc:1397] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:34:50.156101 3869 ts_tablet_manager.cc:1397] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:34:50.156273 3870 log.cc:824] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:50.158253 3870 tablet_bootstrap.cc:492] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903: No bootstrap required, opened a new log
I20260924 10:34:50.158442 3870 ts_tablet_manager.cc:1397] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:34:50.159693 3869 raft_consensus.cc:348] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.159945 3869 raft_consensus.cc:374] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:50.160004 3869 raft_consensus.cc:729] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16ab99afb2443b3b8475f1307036d59, State: Initialized, Role: FOLLOWER
I20260924 10:34:50.159833 3868 raft_consensus.cc:348] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.160075 3868 raft_consensus.cc:374] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:50.160131 3868 raft_consensus.cc:729] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23534aa55cec40a086b9348d5691bd29, State: Initialized, Role: FOLLOWER
I20260924 10:34:50.160528 3869 consensus_queue.cc:260] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.160554 3868 consensus_queue.cc:260] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.161571 3868 ts_tablet_manager.cc:1428] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260924 10:34:50.161603 3869 ts_tablet_manager.cc:1428] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:34:50.161940 3583 heartbeater.cc:503] Master 127.2.193.190:43987 was elected leader, sending a full tablet report...
I20260924 10:34:50.161940 3851 heartbeater.cc:503] Master 127.2.193.190:43987 was elected leader, sending a full tablet report...
I20260924 10:34:50.162180 3870 raft_consensus.cc:348] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.162405 3870 raft_consensus.cc:374] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:50.162463 3870 raft_consensus.cc:729] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad2bf58c95b4a7c8323aaa6f396d903, State: Initialized, Role: FOLLOWER
I20260924 10:34:50.162894 3870 consensus_queue.cc:260] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.163738 3717 heartbeater.cc:503] Master 127.2.193.190:43987 was elected leader, sending a full tablet report...
I20260924 10:34:50.164979 3870 ts_tablet_manager.cc:1428] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 10:34:50.167026 3584 tablet.cc:2369] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:34:50.326372 3852 tablet.cc:2369] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:34:50.373835 3718 tablet.cc:2369] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:34:50.395253 3875 raft_consensus.cc:482] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:50.395509 3875 raft_consensus.cc:504] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.396996 3875 leader_election.cc:288] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23534aa55cec40a086b9348d5691bd29 (127.2.193.129:38509), 3ad2bf58c95b4a7c8323aaa6f396d903 (127.2.193.130:46307)
I20260924 10:34:50.401769 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.397264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38509 (local address 127.2.193.131:45767)
0924 10:34:50.397409 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:50.397430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:50.397548 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:50.397968 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:50.397971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:50.397989 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:50.398117 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.398127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.399106 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.399111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.400436 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.400449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.401328 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.401331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.401433 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.401455 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:34:50.401559 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:34:50.401635 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 10:34:50.401769 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.397566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:45767 (local address 127.2.193.129:38509)
0924 10:34:50.397716 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.397722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.397740 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:34:50.397801 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.397807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.397865 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.397956 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:50.398257 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.399003 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.400585 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.401215 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.401334 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.401388 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.401576 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.401646 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:34:50.402491 3867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.397435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:60159 (local address 127.2.193.130:46307)
0924 10:34:50.397557 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.397562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.398538 (+ 976us) server_negotiation.cc:407] Connection header received
0924 10:34:50.398701 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.398704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.398756 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.398838 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:50.399154 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.399794 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.401193 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.401894 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.402036 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.402111 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.402289 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.402367 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:34:50.402705 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee4d9ab0258418a89a2a724f6b028a5" candidate_uuid: "a16ab99afb2443b3b8475f1307036d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23534aa55cec40a086b9348d5691bd29" is_pre_election: true
I20260924 10:34:50.402837 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.397265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46307 (local address 127.2.193.131:60159)
0924 10:34:50.398430 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:50.398444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:50.398547 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:50.398867 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:50.398870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:50.398892 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:50.399014 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.399020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.399949 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.399953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.401036 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:50.401046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.402048 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.402053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:50.402145 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.402166 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:34:50.402623 (+ 457us) client_negotiation.cc:240] Negotiation successful
0924 10:34:50.402696 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":784,"thread_start_us":128,"threads_started":1}
I20260924 10:34:50.403039 3537 raft_consensus.cc:2454] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16ab99afb2443b3b8475f1307036d59 in term 0.
I20260924 10:34:50.403489 3671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee4d9ab0258418a89a2a724f6b028a5" candidate_uuid: "a16ab99afb2443b3b8475f1307036d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" is_pre_election: true
I20260924 10:34:50.403561 3738 leader_election.cc:302] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23534aa55cec40a086b9348d5691bd29, a16ab99afb2443b3b8475f1307036d59; no voters:
I20260924 10:34:50.403760 3671 raft_consensus.cc:2454] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a16ab99afb2443b3b8475f1307036d59 in term 0.
I20260924 10:34:50.403858 3875 raft_consensus.cc:2781] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:50.403939 3875 raft_consensus.cc:482] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:50.403978 3875 raft_consensus.cc:3037] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:50.405248 3875 raft_consensus.cc:504] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.405551 3875 leader_election.cc:288] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [CANDIDATE]: Term 1 election: Requested vote from peers 23534aa55cec40a086b9348d5691bd29 (127.2.193.129:38509), 3ad2bf58c95b4a7c8323aaa6f396d903 (127.2.193.130:46307)
I20260924 10:34:50.405932 3537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee4d9ab0258418a89a2a724f6b028a5" candidate_uuid: "a16ab99afb2443b3b8475f1307036d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23534aa55cec40a086b9348d5691bd29"
I20260924 10:34:50.406077 3537 raft_consensus.cc:3037] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:50.406175 3671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee4d9ab0258418a89a2a724f6b028a5" candidate_uuid: "a16ab99afb2443b3b8475f1307036d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903"
I20260924 10:34:50.406324 3671 raft_consensus.cc:3037] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:50.407509 3537 raft_consensus.cc:2454] T 4ee4d9ab0258418a89a2a724f6b028a5 P 23534aa55cec40a086b9348d5691bd29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16ab99afb2443b3b8475f1307036d59 in term 1.
I20260924 10:34:50.407713 3671 raft_consensus.cc:2454] T 4ee4d9ab0258418a89a2a724f6b028a5 P 3ad2bf58c95b4a7c8323aaa6f396d903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a16ab99afb2443b3b8475f1307036d59 in term 1.
I20260924 10:34:50.408027 3741 leader_election.cc:302] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad2bf58c95b4a7c8323aaa6f396d903, a16ab99afb2443b3b8475f1307036d59; no voters:
I20260924 10:34:50.408231 3875 raft_consensus.cc:2781] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:50.408450 3875 raft_consensus.cc:686] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [term 1 LEADER]: Becoming Leader. State: Replica: a16ab99afb2443b3b8475f1307036d59, State: Running, Role: LEADER
I20260924 10:34:50.408805 3875 consensus_queue.cc:237] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } }
I20260924 10:34:50.412295 3392 catalog_manager.cc:5495] T 4ee4d9ab0258418a89a2a724f6b028a5 P a16ab99afb2443b3b8475f1307036d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to a16ab99afb2443b3b8475f1307036d59 (127.2.193.131). New cstate: current_term: 1 leader_uuid: "a16ab99afb2443b3b8475f1307036d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16ab99afb2443b3b8475f1307036d59" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23534aa55cec40a086b9348d5691bd29" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ad2bf58c95b4a7c8323aaa6f396d903" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46307 } health_report { overall_health: UNKNOWN } } }
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:34:50.436681 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.424909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.2.193.129:38509)
0924 10:34:50.425056 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.425063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.425083 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:34:50.425134 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.425138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.425234 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.425345 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:50.426331 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.427202 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.428618 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.428918 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.428995 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.432409 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:50.432424 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:50.432443 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:50.432478 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:50.434597 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.435150 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.435157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.435159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.435264 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.435572 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.435577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.435580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.435960 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:50.436221 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.436366 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.436515 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246090, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246090, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:34:50.464936 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.456408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.2.193.129:38509)
0924 10:34:50.456571 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.456575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.456712 (+ 137us) server_negotiation.cc:407] Connection header received
0924 10:34:50.456892 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.456895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.456947 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.457040 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:50.457804 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.458329 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.459177 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.459411 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.459517 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.462105 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:50.462108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:50.462110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:50.462135 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:50.463422 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.463807 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.463809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.463810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.463880 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.464107 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.464109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.464110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.464270 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:50.464363 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.464621 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.464726 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:34:50.490741 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:50.482434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.2.193.129:38509)
0924 10:34:50.482570 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:50.482575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:50.482759 (+ 184us) server_negotiation.cc:407] Connection header received
0924 10:34:50.482944 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:50.482947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:50.482997 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:50.483084 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:50.483790 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.484299 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.485150 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:50.485420 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:50.485528 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:50.487938 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:50.487941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:50.487944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:50.487977 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:50.489312 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.489695 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.489698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.489699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.489768 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:50.489992 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:50.489995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:50.489996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:50.490154 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:50.490248 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:50.490440 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 10:34:50.490539 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 10:34:50.492165 3497 tablet_service.cc:2908] Rejecting scan request for tablet 4ee4d9ab0258418a89a2a724f6b028a5: Uninitialized: clean time has not yet been initialized
I20260924 10:34:50.494135 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 3454
I20260924 10:34:50.499692 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 3588
I20260924 10:34:50.505270 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 3722
I20260924 10:34:50.511018 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 3360
2026-09-24T10:34:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2403 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3892](info): set up 2 sockets
Sep 24 10:34:50 dist-test-slave-8wc7 krb5kdc[3892](info): commencing operation
krb5kdc: starting...
W20260924 10:34:51.549074 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:34:51 dist-test-slave-8wc7 krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:34:51.673053 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:42369
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:42369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:51.786450 3905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:51.786746 3905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:51.786787 3905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:51.791075 3905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:34:51.791146 3905 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:34:51.791163 3905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:51.791183 3905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:51.791200 3905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:34:51.791255 3905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:34:51.796829 3905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:42369
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:42369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:51.798017 3905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:51.799119 3905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:51.807786 3913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:51.807798 3910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:51.807839 3911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:51.807950 3905 server_base.cc:1034] running on GCE node
I20260924 10:34:51.808715 3905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:51.809962 3905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:51.811157 3905 hybrid_clock.cc:648] HybridClock initialized: now 1790246091811142 us; error 28 us; skew 500 ppm
Sep 24 10:34:51 dist-test-slave-8wc7 krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:51.820001 3905 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:34:51.820962 3905 webserver.cc:460] Webserver started at http://127.2.193.190:44411/ using document root <none> and password file <none>
I20260924 10:34:51.821622 3905 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:51.821717 3905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:51.821954 3905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:51.824081 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f"
format_stamp: "Formatted at 2026-09-24 10:34:51 on dist-test-slave-8wc7"
server_key: "d6a73db7e3535909efc3ef1e04ff428b"
server_key_iv: "73e07c540d78a567d440e6341095509f"
server_key_version: "encryptionkey@0"
I20260924 10:34:51.824684 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f"
format_stamp: "Formatted at 2026-09-24 10:34:51 on dist-test-slave-8wc7"
server_key: "d6a73db7e3535909efc3ef1e04ff428b"
server_key_iv: "73e07c540d78a567d440e6341095509f"
server_key_version: "encryptionkey@0"
I20260924 10:34:51.828462 3905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:34:51.831035 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:51.832273 3905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:51.832427 3905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f"
format_stamp: "Formatted at 2026-09-24 10:34:51 on dist-test-slave-8wc7"
server_key: "d6a73db7e3535909efc3ef1e04ff428b"
server_key_iv: "73e07c540d78a567d440e6341095509f"
server_key_version: "encryptionkey@0"
I20260924 10:34:51.832557 3905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:51.848839 3905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:51.851775 3905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:51.852001 3905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:51.860119 3905 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:42369
I20260924 10:34:51.860137 3982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:42369 every 8 connection(s)
I20260924 10:34:51.861120 3905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:34:51.864259 3983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:51.869570 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 3905
I20260924 10:34:51.869700 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:34:51.869992 2822 external_mini_cluster.cc:1428] Setting key fc8d179dc9797323c5e9c5342ed568a1
I20260924 10:34:51.870263 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f: Bootstrap starting.
I20260924 10:34:51.873008 3983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:51.873953 3983 log.cc:824] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:51.876199 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f: No bootstrap required, opened a new log
Sep 24 10:34:51 dist-test-slave-8wc7 krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:51.880584 3983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42369 } }
I20260924 10:34:51.880856 3983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:51.880920 3983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1c6bddf05445ee9fca46eb2ca6d03f, State: Initialized, Role: FOLLOWER
I20260924 10:34:51.881552 3983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42369 } }
I20260924 10:34:51.881745 3983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:34:51.881820 3983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:34:51.881922 3983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:51.883136 3983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42369 } }
I20260924 10:34:51.883543 3983 leader_election.cc:302] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b1c6bddf05445ee9fca46eb2ca6d03f; no voters:
I20260924 10:34:51.883880 3983 leader_election.cc:288] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:34:51.884006 3988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:51.884258 3988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [term 1 LEADER]: Becoming Leader. State: Replica: 9b1c6bddf05445ee9fca46eb2ca6d03f, State: Running, Role: LEADER
I20260924 10:34:51.884562 3988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42369 } }
I20260924 10:34:51.885288 3983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:34:51.886318 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:51.871475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.2.193.190:42369)
0924 10:34:51.871952 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:51.871970 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:51.872013 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:34:51.872849 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:51.872876 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:51.873358 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:51.873738 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:51.874503 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:51.875888 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:51.876780 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:51.877523 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:51.877715 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:51.880289 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:51.880295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:51.880312 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:51.880343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:51.882590 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:51.883016 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:51.883025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:51.883033 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:51.883181 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:51.883486 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:51.883489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:51.883491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:51.884047 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:51.884812 (+ 765us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:51.884962 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:34:51.885285 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":142,"threads_started":1}
I20260924 10:34:51.886402 3990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b1c6bddf05445ee9fca46eb2ca6d03f. Latest consensus state: current_term: 1 leader_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42369 } } }
I20260924 10:34:51.886530 3990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:51.886950 3997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:34:51.887382 3989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1c6bddf05445ee9fca46eb2ca6d03f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42369 } } }
I20260924 10:34:51.887475 3989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:51.890107 3997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:34:51.895489 3997 catalog_manager.cc:1347] Generated new cluster ID: 0ccb7c9d07ca4cacb71ce8a6e0a4a436
I20260924 10:34:51.895576 3997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:34:51.933204 3997 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:34:51.934242 3997 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:34:51.941087 3997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f: Generated new TSK 0
I20260924 10:34:51.941886 3997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:34:51.973057 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:42369
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:52.084450 4009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:52.084735 4009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:52.084780 4009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:52.089143 4009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:52.089227 4009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:52.089268 4009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:34:52.089305 4009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:34:52.094581 4009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:42369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:52.095759 4009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:52.096776 4009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:52.107982 4014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:52.108024 4015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:52.108002 4017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:52.108278 4009 server_base.cc:1034] running on GCE node
I20260924 10:34:52.108719 4009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:52.109433 4009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:52.110589 4009 hybrid_clock.cc:648] HybridClock initialized: now 1790246092110573 us; error 26 us; skew 500 ppm
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:52.113427 4009 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:34:52.114333 4009 webserver.cc:460] Webserver started at http://127.2.193.129:42565/ using document root <none> and password file <none>
I20260924 10:34:52.114892 4009 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:52.115003 4009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:52.115224 4009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:52.117291 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "9f39ee03c04f4aa28b8bd0e710ed5087"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "ce83ad0127df3970640e5e5a0d79e1ea"
server_key_iv: "8014c4ccd8a8828a80a063082d04bd6c"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.117848 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "9f39ee03c04f4aa28b8bd0e710ed5087"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "ce83ad0127df3970640e5e5a0d79e1ea"
server_key_iv: "8014c4ccd8a8828a80a063082d04bd6c"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.121461 4009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:34:52.123720 4024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:52.124881 4009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:52.125013 4009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "9f39ee03c04f4aa28b8bd0e710ed5087"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "ce83ad0127df3970640e5e5a0d79e1ea"
server_key_iv: "8014c4ccd8a8828a80a063082d04bd6c"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.125120 4009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:52.150794 4009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:52.153856 4009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:52.154066 4009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:52.155514 4009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:52.155587 4009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:52.155658 4009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:52.155715 4009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:52.167147 4009 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:35577
I20260924 10:34:52.167222 4139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:35577 every 8 connection(s)
I20260924 10:34:52.168299 4009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:34:52.169690 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.156365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:37539 (local address 127.2.193.190:42369)
0924 10:34:52.156512 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.156517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.157419 (+ 902us) server_negotiation.cc:407] Connection header received
0924 10:34:52.158401 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.158404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.158467 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.158524 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:52.159914 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.160811 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.162238 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.162505 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.162628 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.165766 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:52.165770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:52.165773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:52.165802 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:52.167561 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.168045 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.168048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.168050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.168138 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.168480 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.168484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.168486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.168678 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:52.168790 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.169378 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.169519 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:34:52.169718 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 4009
I20260924 10:34:52.169848 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:34:52.170122 2822 external_mini_cluster.cc:1428] Setting key e4a9872b0df5135a4e2474702753cbc0
I20260924 10:34:52.170547 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.156796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42369 (local address 127.2.193.129:37539)
0924 10:34:52.157262 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.157313 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.158147 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.158855 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.158862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.159175 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:52.159725 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.159744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.160956 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.160960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.162111 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.162120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.162673 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.162678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.162856 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.163515 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:52.163535 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:52.165429 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:52.167668 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.167672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.167683 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.167926 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.168225 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.168227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.168229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.168385 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.168786 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:52.168792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:52.169072 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.169323 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.169669 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":133,"threads_started":1}
I20260924 10:34:52.172061 4140 heartbeater.cc:347] Connected to a master server at 127.2.193.190:42369
I20260924 10:34:52.172214 4140 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:52.172770 4140 heartbeater.cc:511] Master 127.2.193.190:42369 requested a full tablet report, sending...
I20260924 10:34:52.174487 3935 ts_manager.cc:194] Registered new tserver with Master: 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129:35577)
I20260924 10:34:52.175927 3935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:37539
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:34:52.182209 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.174076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:49499 (local address 127.2.193.190:42369)
0924 10:34:52.174185 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.174189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.174951 (+ 762us) server_negotiation.cc:407] Connection header received
0924 10:34:52.175064 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.175067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.175104 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.175169 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:52.175730 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.176440 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.177300 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.177736 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.177867 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.178955 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:52.178961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:52.178965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:52.178995 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:52.180602 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.181003 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.181007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.181008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.181090 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.181399 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.181404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.181406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.181581 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:52.181689 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.181937 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.182058 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":46}
I20260924 10:34:52.182214 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.174018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42369 (local address 127.2.193.129:49499)
0924 10:34:52.174866 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.174881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.174984 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.175212 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.175218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.175457 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:52.175609 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.175619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.176552 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.176555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.177130 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.177138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.177851 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.177854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.177922 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.178491 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:52.178503 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:52.178819 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:52.180721 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.180723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.180725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.180906 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.181218 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.181225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.181227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.181297 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.181678 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:52.181681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:52.181775 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.181877 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.182010 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:34:52.200829 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:42369
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:52.313089 4149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:52.313437 4149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:52.313524 4149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:52.317498 4149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:52.317582 4149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:52.317654 4149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:34:52.317731 4149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:34:52.322906 4149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:42369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:52.323930 4149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:52.324929 4149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:52.334532 4154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:52.334532 4155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:52.334542 4157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:52.334568 4149 server_base.cc:1034] running on GCE node
I20260924 10:34:52.335167 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:52.335723 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:52.336874 4149 hybrid_clock.cc:648] HybridClock initialized: now 1790246092336825 us; error 59 us; skew 500 ppm
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:52.339519 4149 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:34:52.340396 4149 webserver.cc:460] Webserver started at http://127.2.193.130:39059/ using document root <none> and password file <none>
I20260924 10:34:52.340936 4149 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:52.341032 4149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:52.341311 4149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:52.343314 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "df2cace2800243a28690e68c290de9cd"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "32c1e6147ba0746c76da49e886777b58"
server_key_iv: "d43e8ae580de427f532f161f3cdba901"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.343866 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "df2cace2800243a28690e68c290de9cd"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "32c1e6147ba0746c76da49e886777b58"
server_key_iv: "d43e8ae580de427f532f161f3cdba901"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.347513 4149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 10:34:52.349848 4164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:52.350986 4149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 10:34:52.351110 4149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "df2cace2800243a28690e68c290de9cd"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "32c1e6147ba0746c76da49e886777b58"
server_key_iv: "d43e8ae580de427f532f161f3cdba901"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.351214 4149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:52.365154 4149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:52.367898 4149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:52.368099 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:52.369570 4149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:52.369635 4149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:52.369699 4149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:52.369755 4149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:52.380409 4149 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:35845
I20260924 10:34:52.380563 4279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:35845 every 8 connection(s)
I20260924 10:34:52.381408 4149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:34:52.383276 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.370269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:52069 (local address 127.2.193.190:42369)
0924 10:34:52.370405 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.370408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.372056 (+ 1648us) server_negotiation.cc:407] Connection header received
0924 10:34:52.373303 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.373307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.373362 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.373470 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:52.374670 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.375315 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.376808 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.377250 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.377372 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.379676 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:52.379679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:52.379682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:52.379708 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:52.381281 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.381804 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.381809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.381811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.381903 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.382306 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.382309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.382310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.382551 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:52.382636 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.383063 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.383161 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:34:52.384054 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.371313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42369 (local address 127.2.193.130:52069)
0924 10:34:52.371877 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.371934 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.373043 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.373644 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.373653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.374072 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:52.374528 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.374541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.375437 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.375440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.376665 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.376682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.377405 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.377409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.377546 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.378121 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:52.378137 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:52.379516 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:52.381388 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.381393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.381407 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.381699 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.382021 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.382025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.382030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.382202 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.382644 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:52.382648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:52.382855 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.383015 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.383245 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":144,"threads_started":1}
I20260924 10:34:52.385367 4280 heartbeater.cc:347] Connected to a master server at 127.2.193.190:42369
I20260924 10:34:52.385516 4280 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:52.386233 4280 heartbeater.cc:511] Master 127.2.193.190:42369 requested a full tablet report, sending...
I20260924 10:34:52.386955 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 4149
I20260924 10:34:52.387059 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:34:52.387368 2822 external_mini_cluster.cc:1428] Setting key 18ebcc3e518a5e465cf063c2ac5d5172
I20260924 10:34:52.388079 3937 ts_manager.cc:194] Registered new tserver with Master: df2cace2800243a28690e68c290de9cd (127.2.193.130:35845)
I20260924 10:34:52.388731 3937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:52069
I20260924 10:34:52.395342 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.387575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42369 (local address 127.2.193.130:43495)
0924 10:34:52.387839 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.387857 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.387948 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.388226 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.388229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.388468 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:52.388614 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.388621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.389687 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.389693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.390690 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.390702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.391144 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.391149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.391254 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.391793 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:52.391809 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:52.392132 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:52.393828 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.393832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.393834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.394035 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.394311 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.394316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.394318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.394403 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.394806 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:52.394809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:52.394906 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.395009 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.395150 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260924 10:34:52.395335 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.387629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:43495 (local address 127.2.193.190:42369)
0924 10:34:52.387750 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.387754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.387928 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:34:52.388043 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.388046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.388086 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.388176 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:52.388752 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.389531 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.390824 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.391027 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.391109 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.392263 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:52.392267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:52.392270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:52.392292 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:52.393702 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.394132 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.394136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.394137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.394219 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.394495 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.394499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.394500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.394711 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:52.394819 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.395074 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.395180 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:34:52.416234 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:42369
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:52.528333 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:52.528646 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:52.528726 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:52.532941 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:52.533011 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:52.533052 4289 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:34:52.533138 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:34:52.538666 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:42369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:52.539744 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:52.540761 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:52.550347 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:52.550369 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:52.550348 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:52.550633 4289 server_base.cc:1034] running on GCE node
I20260924 10:34:52.551108 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:52.551769 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:52.552966 4289 hybrid_clock.cc:648] HybridClock initialized: now 1790246092552931 us; error 55 us; skew 500 ppm
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:52.555650 4289 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:34:52.556530 4289 webserver.cc:460] Webserver started at http://127.2.193.131:38243/ using document root <none> and password file <none>
I20260924 10:34:52.557089 4289 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:52.557178 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:52.557427 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:52.559461 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "11450cccd0224021967ddcb1ae30ecf7"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "2379ee5c4cd435f6ba03be388891d1b0"
server_key_iv: "17f2b7c50dfec00ce8b734c5897f2d3e"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.560014 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "11450cccd0224021967ddcb1ae30ecf7"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "2379ee5c4cd435f6ba03be388891d1b0"
server_key_iv: "17f2b7c50dfec00ce8b734c5897f2d3e"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.563988 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 10:34:52.566434 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:52.567579 4289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:34:52.567713 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "11450cccd0224021967ddcb1ae30ecf7"
format_stamp: "Formatted at 2026-09-24 10:34:52 on dist-test-slave-8wc7"
server_key: "2379ee5c4cd435f6ba03be388891d1b0"
server_key_iv: "17f2b7c50dfec00ce8b734c5897f2d3e"
server_key_version: "encryptionkey@0"
I20260924 10:34:52.567822 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:52.579420 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:52.582108 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:52.582319 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:52.583755 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:52.583822 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:52.583887 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:52.583922 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:52.595024 4289 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:44629
I20260924 10:34:52.595127 4419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:44629 every 8 connection(s)
I20260924 10:34:52.596019 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:34:52.597829 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.584500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:36949 (local address 127.2.193.190:42369)
0924 10:34:52.584739 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.584746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.586215 (+ 1469us) server_negotiation.cc:407] Connection header received
0924 10:34:52.587764 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.587771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.587840 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.587962 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:52.589122 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.589713 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.591005 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.591223 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.591337 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.594116 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:52.594119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:52.594121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:52.594146 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:52.595691 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.596273 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.596277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.596278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.596372 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.596755 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.596758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.596759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.597010 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:52.597115 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.597614 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.597711 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260924 10:34:52.598621 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.585453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42369 (local address 127.2.193.131:36949)
0924 10:34:52.586022 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.586084 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.587508 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.588143 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.588150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.588496 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:52.588972 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.588985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.589837 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.589841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.590866 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.590875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.591379 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.591383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.591550 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.592143 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:52.592158 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:52.593953 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:52.595803 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.595808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.595821 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.596153 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.596476 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.596480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.596482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.596638 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.597121 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:52.597126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:52.597407 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.597565 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.597798 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":124,"threads_started":1}
I20260924 10:34:52.599745 4420 heartbeater.cc:347] Connected to a master server at 127.2.193.190:42369
I20260924 10:34:52.599903 4420 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:52.600394 4420 heartbeater.cc:511] Master 127.2.193.190:42369 requested a full tablet report, sending...
I20260924 10:34:52.601413 3935 ts_manager.cc:194] Registered new tserver with Master: 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131:44629)
I20260924 10:34:52.602200 3935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:36949
I20260924 10:34:52.602685 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 4289
I20260924 10:34:52.602811 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:34:52.603091 2822 external_mini_cluster.cc:1428] Setting key 0953c47666fe1fdc90299412a2bbfb9a
I20260924 10:34:52.605062 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:34:52.609311 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.601783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:59487 (local address 127.2.193.190:42369)
0924 10:34:52.601897 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.601900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.602123 (+ 223us) server_negotiation.cc:407] Connection header received
0924 10:34:52.602224 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.602226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.602258 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.602353 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:52.602843 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.603403 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.604492 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.604794 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.604891 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.606047 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:52.606053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:52.606059 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:52.606089 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:52.607546 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.608025 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.608027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.608028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.608094 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.608378 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.608383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.608385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.608561 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:52.608731 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.609014 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.609118 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:34:52.609319 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.601710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42369 (local address 127.2.193.131:59487)
0924 10:34:52.602045 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.602058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.602148 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.602361 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.602364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.602588 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:52.602702 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.602710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.603529 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.603532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.604385 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.604395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.604912 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.604916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.605013 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.605588 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:52.605602 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:52.605913 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:52.607651 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.607656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.607658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.607919 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.608192 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:52.608195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:52.608197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:52.608285 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:52.608714 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:52.608718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:52.608811 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.608954 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.609106 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:34:52 dist-test-slave-8wc7 krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:52.630574 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.622808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.2.193.190:42369)
0924 10:34:52.622934 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.622939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.622970 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:34:52.623010 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.623012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.623049 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.623119 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:52.623844 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.624347 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.625107 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.625390 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.625468 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.627657 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:52.627660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:52.627662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:52.627684 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:52.629134 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.629577 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.629580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.629581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.629649 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:52.629858 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:52.629860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:52.629861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:52.630023 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:52.630117 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.630303 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.630388 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:34:52.633452 3935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:34:52.636324 3935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:52.650458 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.644433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35577 (local address 127.0.0.1:54758)
0924 10:34:52.644850 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.644895 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.645029 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.645889 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.645894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.645915 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:52.646040 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.646047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.647575 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.647578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.648839 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.648847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.649871 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.649875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.650000 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.650039 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.650147 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.650275 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":8,"thread_start_us":90,"threads_started":1}
I20260924 10:34:52.650660 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.644549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.2.193.129:35577)
0924 10:34:52.645385 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.645402 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.645428 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:34:52.645515 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.645521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.645722 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.646187 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:52.646315 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.647447 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.648956 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.649736 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.649849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.650025 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.650409 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.650517 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":76,"threads_started":1}
I20260924 10:34:52.650650 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.644506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:35845 (local address 127.0.0.1:38412)
0924 10:34:52.645269 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.645281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.645358 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.646022 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.646025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.646038 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:52.646134 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.646140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.647613 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.647638 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.649121 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.649131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.650174 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.650178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.650350 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.650395 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.650462 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.650527 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":133,"thread_start_us":111,"threads_started":1}
I20260924 10:34:52.651088 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.645881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.2.193.131:44629)
0924 10:34:52.646268 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.646276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.646292 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:34:52.646349 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.646353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.646482 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.646630 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:52.646802 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.647565 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.649309 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.649884 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.649987 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.650145 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.650838 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.650939 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260924 10:34:52.651296 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.645494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:44629 (local address 127.0.0.1:57348)
0924 10:34:52.646004 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.646019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.646102 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.646608 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.646611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.646627 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:52.646710 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.646718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.647673 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.647676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.649142 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.649149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.650201 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.650204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.650368 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.650411 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.650490 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.651026 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":381,"thread_start_us":89,"threads_started":1}
I20260924 10:34:52.651826 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.644876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.2.193.130:35845)
0924 10:34:52.645203 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.645217 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.645445 (+ 228us) server_negotiation.cc:407] Connection header received
0924 10:34:52.645515 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.645521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.645722 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.645871 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:52.646244 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.647456 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.649256 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.650040 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.650163 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.650318 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.650877 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.650981 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260924 10:34:52.653364 4074 tablet_service.cc:1467] Processing CreateTablet for tablet f76248fdf71e472094eabe7963b9704a (DEFAULT_TABLE table=test-table [id=e5f85b0cdc704b8a899256c10ee547d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:52.653512 4354 tablet_service.cc:1467] Processing CreateTablet for tablet f76248fdf71e472094eabe7963b9704a (DEFAULT_TABLE table=test-table [id=e5f85b0cdc704b8a899256c10ee547d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:52.654392 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76248fdf71e472094eabe7963b9704a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:52.654399 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76248fdf71e472094eabe7963b9704a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:52.660817 4444 tablet_bootstrap.cc:492] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Bootstrap starting.
I20260924 10:34:52.660888 4443 tablet_bootstrap.cc:492] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Bootstrap starting.
I20260924 10:34:52.662962 4444 tablet_bootstrap.cc:654] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:52.663017 4443 tablet_bootstrap.cc:654] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:52.662612 4214 tablet_service.cc:1467] Processing CreateTablet for tablet f76248fdf71e472094eabe7963b9704a (DEFAULT_TABLE table=test-table [id=e5f85b0cdc704b8a899256c10ee547d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:52.663681 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76248fdf71e472094eabe7963b9704a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:52.663774 4443 log.cc:824] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:52.663762 4444 log.cc:824] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:52.665661 4443 tablet_bootstrap.cc:492] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: No bootstrap required, opened a new log
I20260924 10:34:52.665843 4443 ts_tablet_manager.cc:1397] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:34:52.666075 4444 tablet_bootstrap.cc:492] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: No bootstrap required, opened a new log
I20260924 10:34:52.666251 4444 ts_tablet_manager.cc:1397] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:34:52.668992 4444 raft_consensus.cc:348] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.669261 4444 raft_consensus.cc:374] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:52.669322 4444 raft_consensus.cc:729] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f39ee03c04f4aa28b8bd0e710ed5087, State: Initialized, Role: FOLLOWER
I20260924 10:34:52.669792 4444 consensus_queue.cc:260] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.669836 4443 raft_consensus.cc:348] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.670089 4443 raft_consensus.cc:374] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:52.670151 4443 raft_consensus.cc:729] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11450cccd0224021967ddcb1ae30ecf7, State: Initialized, Role: FOLLOWER
I20260924 10:34:52.670570 4443 consensus_queue.cc:260] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.670802 4444 ts_tablet_manager.cc:1428] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260924 10:34:52.671206 4140 heartbeater.cc:503] Master 127.2.193.190:42369 was elected leader, sending a full tablet report...
I20260924 10:34:52.671538 4443 ts_tablet_manager.cc:1428] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:34:52.671749 4449 tablet_bootstrap.cc:492] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Bootstrap starting.
I20260924 10:34:52.671875 4420 heartbeater.cc:503] Master 127.2.193.190:42369 was elected leader, sending a full tablet report...
I20260924 10:34:52.674366 4449 tablet_bootstrap.cc:654] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:52.675282 4449 log.cc:824] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:52.677155 4449 tablet_bootstrap.cc:492] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: No bootstrap required, opened a new log
I20260924 10:34:52.677383 4449 ts_tablet_manager.cc:1397] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260924 10:34:52.680197 4449 raft_consensus.cc:348] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.680433 4449 raft_consensus.cc:374] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:52.680495 4449 raft_consensus.cc:729] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2cace2800243a28690e68c290de9cd, State: Initialized, Role: FOLLOWER
I20260924 10:34:52.680922 4449 consensus_queue.cc:260] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.682120 4280 heartbeater.cc:503] Master 127.2.193.190:42369 was elected leader, sending a full tablet report...
I20260924 10:34:52.682425 4449 ts_tablet_manager.cc:1428] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260924 10:34:52.798234 4447 raft_consensus.cc:482] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:52.798483 4447 raft_consensus.cc:504] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.799638 4447 leader_election.cc:288] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2cace2800243a28690e68c290de9cd (127.2.193.130:35845), 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131:44629)
I20260924 10:34:52.803561 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.800033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:38273 (local address 127.2.193.130:35845)
0924 10:34:52.800189 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.800194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.800306 (+ 112us) server_negotiation.cc:407] Connection header received
0924 10:34:52.800478 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.800481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.800531 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.800623 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:52.800908 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.801526 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.802566 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.802981 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.803111 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.803165 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.803345 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.803414 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:34:52.803586 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.800059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:43317 (local address 127.2.193.131:44629)
0924 10:34:52.800196 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.800207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.800531 (+ 324us) server_negotiation.cc:407] Connection header received
0924 10:34:52.800695 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.800699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.800749 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.800833 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:52.801078 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.801658 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.802632 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.803017 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.803144 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.803197 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.803372 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.803441 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:34:52.803627 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.799899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:35845 (local address 127.2.193.129:38273)
0924 10:34:52.800190 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.800207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.800356 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.800646 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.800649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.800666 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:52.800796 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.800801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.801643 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.801647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.802438 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.802446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.803105 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.803109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.803202 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.803225 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.803328 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.803414 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260924 10:34:52.803632 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.799899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:44629 (local address 127.2.193.129:43317)
0924 10:34:52.800440 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:52.800451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:52.800594 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:52.800844 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:52.800847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:52.800862 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:52.800977 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.800983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.801770 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.801775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.802507 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:52.802514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.803135 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.803138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:52.803241 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.803269 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:34:52.803355 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:34:52.803450 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":341,"thread_start_us":83,"threads_started":1}
I20260924 10:34:52.804241 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76248fdf71e472094eabe7963b9704a" candidate_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2cace2800243a28690e68c290de9cd" is_pre_election: true
I20260924 10:34:52.804246 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76248fdf71e472094eabe7963b9704a" candidate_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11450cccd0224021967ddcb1ae30ecf7" is_pre_election: true
I20260924 10:34:52.804548 4234 raft_consensus.cc:2454] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f39ee03c04f4aa28b8bd0e710ed5087 in term 0.
I20260924 10:34:52.804564 4374 raft_consensus.cc:2454] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f39ee03c04f4aa28b8bd0e710ed5087 in term 0.
I20260924 10:34:52.805055 4028 leader_election.cc:302] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11450cccd0224021967ddcb1ae30ecf7, 9f39ee03c04f4aa28b8bd0e710ed5087, df2cace2800243a28690e68c290de9cd; no voters:
I20260924 10:34:52.805352 4447 raft_consensus.cc:2781] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:52.805436 4447 raft_consensus.cc:482] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:52.805478 4447 raft_consensus.cc:3037] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:52.806517 4447 raft_consensus.cc:504] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.806855 4447 leader_election.cc:288] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [CANDIDATE]: Term 1 election: Requested vote from peers df2cace2800243a28690e68c290de9cd (127.2.193.130:35845), 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131:44629)
I20260924 10:34:52.807202 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76248fdf71e472094eabe7963b9704a" candidate_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11450cccd0224021967ddcb1ae30ecf7"
I20260924 10:34:52.807207 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76248fdf71e472094eabe7963b9704a" candidate_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2cace2800243a28690e68c290de9cd"
I20260924 10:34:52.807351 4234 raft_consensus.cc:3037] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:52.807320 4374 raft_consensus.cc:3037] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:52.808382 4374 raft_consensus.cc:2454] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f39ee03c04f4aa28b8bd0e710ed5087 in term 1.
I20260924 10:34:52.808696 4028 leader_election.cc:302] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11450cccd0224021967ddcb1ae30ecf7, 9f39ee03c04f4aa28b8bd0e710ed5087; no voters:
I20260924 10:34:52.808861 4234 raft_consensus.cc:2454] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f39ee03c04f4aa28b8bd0e710ed5087 in term 1.
I20260924 10:34:52.808881 4447 raft_consensus.cc:2781] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:52.809119 4447 raft_consensus.cc:686] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 LEADER]: Becoming Leader. State: Replica: 9f39ee03c04f4aa28b8bd0e710ed5087, State: Running, Role: LEADER
I20260924 10:34:52.809477 4447 consensus_queue.cc:237] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.812314 3935 catalog_manager.cc:5495] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } health_report { overall_health: UNKNOWN } } }
I20260924 10:34:52.844172 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:52.840319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.2.193.129:35577)
0924 10:34:52.840467 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:52.840472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:52.840488 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:34:52.840726 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:52.840729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:52.840782 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:52.840887 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:52.841159 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.841689 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.842610 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:52.842831 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:52.842944 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:52.843288 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:52.843727 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:52.843851 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:52.843939 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 10:34:52.844008 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 10:34:52.847702 4421 tablet.cc:2369] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:34:52.850533 4234 raft_consensus.cc:1261] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Refusing update from remote peer 9f39ee03c04f4aa28b8bd0e710ed5087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:52.850531 4374 raft_consensus.cc:1261] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Refusing update from remote peer 9f39ee03c04f4aa28b8bd0e710ed5087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:52.851249 4454 consensus_queue.cc:1035] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:52.851439 4447 consensus_queue.cc:1035] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:52.859014 4458 mvcc.cc:205] Tried to move back new op lower bound from 7332847996308492288 to 7332847996150333440. Current Snapshot: MvccSnapshot[applied={T|T < 7332847996308492288}]
I20260924 10:34:52.863960 4459 mvcc.cc:205] Tried to move back new op lower bound from 7332847996308492288 to 7332847996150333440. Current Snapshot: MvccSnapshot[applied={T|T < 7332847996308492288 or (T in {7332847996308492288})}]
I20260924 10:34:52.872025 3935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:52.872264 3935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:52.875828 3935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f: Sending DeleteTablet for 3 replicas of tablet f76248fdf71e472094eabe7963b9704a
I20260924 10:34:52.877635 4214 tablet_service.cc:1514] Processing DeleteTablet for tablet f76248fdf71e472094eabe7963b9704a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:52 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:38412
I20260924 10:34:52.878213 4466 tablet_replica.cc:330] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: stopping tablet replica
I20260924 10:34:52.878379 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet f76248fdf71e472094eabe7963b9704a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:52 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:54758
I20260924 10:34:52.878578 4466 raft_consensus.cc:2229] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:52.878945 4466 raft_consensus.cc:2258] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:52.879081 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet f76248fdf71e472094eabe7963b9704a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:52 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:57348
I20260924 10:34:52.879721 4467 tablet_replica.cc:330] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: stopping tablet replica
I20260924 10:34:52.879715 3935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:34:52.880048 4467 raft_consensus.cc:2229] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 LEADER]: Raft consensus shutting down.
W20260924 10:34:52.880187 3935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:52.880466 4467 raft_consensus.cc:2258] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:52.880854 4468 tablet_replica.cc:330] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: stopping tablet replica
I20260924 10:34:52.880937 4466 ts_tablet_manager.cc:1905] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:52.881155 4468 raft_consensus.cc:2229] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:52.881506 4468 raft_consensus.cc:2258] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:52.882212 4467 ts_tablet_manager.cc:1905] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:52.883778 4466 ts_tablet_manager.cc:1918] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:34:52.883889 4466 log.cc:1196] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/f76248fdf71e472094eabe7963b9704a
I20260924 10:34:52.884219 4466 ts_tablet_manager.cc:1939] T f76248fdf71e472094eabe7963b9704a P df2cace2800243a28690e68c290de9cd: Deleting consensus metadata
I20260924 10:34:52.885948 4468 ts_tablet_manager.cc:1905] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:52.886257 3921 catalog_manager.cc:4841] TS df2cace2800243a28690e68c290de9cd (127.2.193.130:35845): tablet f76248fdf71e472094eabe7963b9704a (table test-table [id=e5f85b0cdc704b8a899256c10ee547d0]) successfully deleted
I20260924 10:34:52.886901 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 344f3f29349d40bbb7139c9ada2cc8b1 (DEFAULT_TABLE table=test-table [id=d3e5ed0925834d72b56331d558482781]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:52.887180 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344f3f29349d40bbb7139c9ada2cc8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:52.887328 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 344f3f29349d40bbb7139c9ada2cc8b1 (DEFAULT_TABLE table=test-table [id=d3e5ed0925834d72b56331d558482781]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:52.887596 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344f3f29349d40bbb7139c9ada2cc8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:52.888270 4467 ts_tablet_manager.cc:1918] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:34:52.889134 4468 ts_tablet_manager.cc:1918] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:34:52.889257 4468 log.cc:1196] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/f76248fdf71e472094eabe7963b9704a
I20260924 10:34:52.889374 4467 log.cc:1196] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/f76248fdf71e472094eabe7963b9704a
I20260924 10:34:52.889607 4468 ts_tablet_manager.cc:1939] T f76248fdf71e472094eabe7963b9704a P 11450cccd0224021967ddcb1ae30ecf7: Deleting consensus metadata
I20260924 10:34:52.889837 4449 tablet_bootstrap.cc:492] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Bootstrap starting.
I20260924 10:34:52.890072 4467 ts_tablet_manager.cc:1939] T f76248fdf71e472094eabe7963b9704a P 9f39ee03c04f4aa28b8bd0e710ed5087: Deleting consensus metadata
I20260924 10:34:52.890816 4449 tablet_bootstrap.cc:654] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:52.891000 3924 catalog_manager.cc:4841] TS 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131:44629): tablet f76248fdf71e472094eabe7963b9704a (table test-table [id=e5f85b0cdc704b8a899256c10ee547d0]) successfully deleted
I20260924 10:34:52.890197 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 344f3f29349d40bbb7139c9ada2cc8b1 (DEFAULT_TABLE table=test-table [id=d3e5ed0925834d72b56331d558482781]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:52.891453 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344f3f29349d40bbb7139c9ada2cc8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:52.891932 4443 tablet_bootstrap.cc:492] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: Bootstrap starting.
I20260924 10:34:52.892582 4449 tablet_bootstrap.cc:492] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: No bootstrap required, opened a new log
I20260924 10:34:52.892673 4449 ts_tablet_manager.cc:1397] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260924 10:34:52.892877 4443 tablet_bootstrap.cc:654] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:52.893219 4449 raft_consensus.cc:348] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.893333 4449 raft_consensus.cc:374] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:52.893362 4449 raft_consensus.cc:729] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2cace2800243a28690e68c290de9cd, State: Initialized, Role: FOLLOWER
I20260924 10:34:52.893548 4449 consensus_queue.cc:260] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.893998 4444 tablet_bootstrap.cc:492] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Bootstrap starting.
I20260924 10:34:52.894922 4444 tablet_bootstrap.cc:654] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:52.894897 3924 catalog_manager.cc:4841] TS 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129:35577): tablet f76248fdf71e472094eabe7963b9704a (table test-table [id=e5f85b0cdc704b8a899256c10ee547d0]) successfully deleted
I20260924 10:34:52.895574 4449 ts_tablet_manager.cc:1428] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 10:34:52.896229 4444 tablet_bootstrap.cc:492] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: No bootstrap required, opened a new log
I20260924 10:34:52.896317 4444 ts_tablet_manager.cc:1397] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 10:34:52.896987 4444 raft_consensus.cc:348] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.897107 4444 raft_consensus.cc:374] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:52.897140 4444 raft_consensus.cc:729] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f39ee03c04f4aa28b8bd0e710ed5087, State: Initialized, Role: FOLLOWER
I20260924 10:34:52.897291 4444 consensus_queue.cc:260] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.897397 4443 tablet_bootstrap.cc:492] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: No bootstrap required, opened a new log
I20260924 10:34:52.897492 4443 ts_tablet_manager.cc:1397] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260924 10:34:52.897536 4444 ts_tablet_manager.cc:1428] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:52.897912 4443 raft_consensus.cc:348] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.898051 4443 raft_consensus.cc:374] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:52.898085 4443 raft_consensus.cc:729] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11450cccd0224021967ddcb1ae30ecf7, State: Initialized, Role: FOLLOWER
I20260924 10:34:52.898218 4443 consensus_queue.cc:260] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:52.898471 4443 ts_tablet_manager.cc:1428] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260924 10:34:52.919936 4141 tablet.cc:2369] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:34:53.116287 4451 raft_consensus.cc:482] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:53.116454 4451 raft_consensus.cc:504] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.117727 4451 leader_election.cc:288] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129:35577), 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131:44629)
I20260924 10:34:53.121763 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.118461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:38317 (local address 127.2.193.131:44629)
0924 10:34:53.118576 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.118582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.118625 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:34:53.118758 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.118763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.118813 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.118887 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:53.119139 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.119630 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.120703 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.121153 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.121307 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.121367 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.121540 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.121603 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:34:53.121807 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.118077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:56797 (local address 127.2.193.129:35577)
0924 10:34:53.118235 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.118241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.118324 (+ 83us) server_negotiation.cc:407] Connection header received
0924 10:34:53.118535 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.118538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.118593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.118650 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:53.119085 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.119579 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.120758 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.121167 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.121346 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.121399 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.121599 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.121687 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:34:53.121870 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.118321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:44629 (local address 127.2.193.130:38317)
0924 10:34:53.118552 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.118563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.118654 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.118902 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.118905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.118917 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:53.119021 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.119025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.119737 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.119743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.120571 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.120585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.121298 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.121302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.121407 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.121435 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.121525 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.121600 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260924 10:34:53.122030 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.117953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35577 (local address 127.2.193.130:56797)
0924 10:34:53.118204 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.118218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.118365 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.118815 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.118818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.118838 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:53.118969 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.118978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.119692 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.119697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.120640 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.120654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.121357 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.121360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.121469 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.121497 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.121584 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.121658 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":87,"thread_start_us":72,"threads_started":1}
I20260924 10:34:53.122314 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344f3f29349d40bbb7139c9ada2cc8b1" candidate_uuid: "df2cace2800243a28690e68c290de9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11450cccd0224021967ddcb1ae30ecf7" is_pre_election: true
I20260924 10:34:53.122484 4374 raft_consensus.cc:2454] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2cace2800243a28690e68c290de9cd in term 0.
I20260924 10:34:53.122776 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344f3f29349d40bbb7139c9ada2cc8b1" candidate_uuid: "df2cace2800243a28690e68c290de9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" is_pre_election: true
I20260924 10:34:53.122848 4168 leader_election.cc:302] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11450cccd0224021967ddcb1ae30ecf7, df2cace2800243a28690e68c290de9cd; no voters:
I20260924 10:34:53.123025 4094 raft_consensus.cc:2454] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df2cace2800243a28690e68c290de9cd in term 0.
I20260924 10:34:53.123093 4451 raft_consensus.cc:2781] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:53.123154 4451 raft_consensus.cc:482] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:53.123201 4451 raft_consensus.cc:3037] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:53.124219 4451 raft_consensus.cc:504] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.124564 4451 leader_election.cc:288] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [CANDIDATE]: Term 1 election: Requested vote from peers 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129:35577), 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131:44629)
I20260924 10:34:53.124935 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344f3f29349d40bbb7139c9ada2cc8b1" candidate_uuid: "df2cace2800243a28690e68c290de9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087"
I20260924 10:34:53.125017 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344f3f29349d40bbb7139c9ada2cc8b1" candidate_uuid: "df2cace2800243a28690e68c290de9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11450cccd0224021967ddcb1ae30ecf7"
I20260924 10:34:53.125058 4094 raft_consensus.cc:3037] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:53.125135 4374 raft_consensus.cc:3037] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:53.126319 4094 raft_consensus.cc:2454] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2cace2800243a28690e68c290de9cd in term 1.
I20260924 10:34:53.126319 4374 raft_consensus.cc:2454] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df2cace2800243a28690e68c290de9cd in term 1.
I20260924 10:34:53.126616 4168 leader_election.cc:302] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f39ee03c04f4aa28b8bd0e710ed5087, df2cace2800243a28690e68c290de9cd; no voters:
I20260924 10:34:53.126822 4451 raft_consensus.cc:2781] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:53.127024 4451 raft_consensus.cc:686] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 1 LEADER]: Becoming Leader. State: Replica: df2cace2800243a28690e68c290de9cd, State: Running, Role: LEADER
I20260924 10:34:53.127328 4451 consensus_queue.cc:237] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.129881 3937 catalog_manager.cc:5495] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd reported cstate change: term changed from 0 to 1, leader changed from <none> to df2cace2800243a28690e68c290de9cd (127.2.193.130). New cstate: current_term: 1 leader_uuid: "df2cace2800243a28690e68c290de9cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } health_report { overall_health: UNKNOWN } } }
W20260924 10:34:53.134786 4281 tablet.cc:2369] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:34:53 dist-test-slave-8wc7 krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:53.219231 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.210927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42369 (local address 127.0.0.1:54126)
0924 10:34:53.211227 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.211240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.211337 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.211576 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.211578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.211984 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:53.212078 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.212084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.212810 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.212812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.213403 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.213411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.213821 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.213823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.213882 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.214508 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:53.214521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:53.216025 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:53.217690 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:53.217698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:53.217703 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:53.217947 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:53.218241 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:53.218244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:53.218245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:53.218316 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:53.218686 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:53.218692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:53.218799 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.218890 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.219018 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
I20260924 10:34:53.219231 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.211007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.2.193.190:42369)
0924 10:34:53.211297 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.211303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.211318 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:34:53.211420 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.211422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.211460 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.211539 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:53.212183 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.212721 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.213501 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.213715 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.213792 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.216157 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:53.216159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:53.216161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:53.216181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:53.217586 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:53.218064 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:53.218067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:53.218068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:53.218154 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:53.218394 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:53.218396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:53.218397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:53.218595 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:53.218675 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.218976 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.219082 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":131,"threads_started":1}
I20260924 10:34:53.223320 3937 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:54126:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:34:53.224089 3937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:53.228963 4074 tablet_service.cc:1467] Processing CreateTablet for tablet a6a34956e2554fec9087a911f38978fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70f72778fa604c698cd113b9489be11b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:34:53.229094 4214 tablet_service.cc:1467] Processing CreateTablet for tablet a6a34956e2554fec9087a911f38978fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70f72778fa604c698cd113b9489be11b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:34:53.229449 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a34956e2554fec9087a911f38978fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:53.229451 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a34956e2554fec9087a911f38978fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:53.232412 4444 tablet_bootstrap.cc:492] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: Bootstrap starting.
I20260924 10:34:53.232417 4449 tablet_bootstrap.cc:492] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: Bootstrap starting.
I20260924 10:34:53.232816 4354 tablet_service.cc:1467] Processing CreateTablet for tablet a6a34956e2554fec9087a911f38978fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70f72778fa604c698cd113b9489be11b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:34:53.233104 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a34956e2554fec9087a911f38978fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:53.233359 4444 tablet_bootstrap.cc:654] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:53.233359 4449 tablet_bootstrap.cc:654] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:53.234663 4444 tablet_bootstrap.cc:492] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: No bootstrap required, opened a new log
I20260924 10:34:53.234724 4449 tablet_bootstrap.cc:492] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: No bootstrap required, opened a new log
I20260924 10:34:53.234759 4444 ts_tablet_manager.cc:1397] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:53.234789 4449 ts_tablet_manager.cc:1397] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:53.235587 4444 raft_consensus.cc:348] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.235641 4449 raft_consensus.cc:348] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.235721 4444 raft_consensus.cc:374] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:53.235735 4449 raft_consensus.cc:374] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:53.235766 4444 raft_consensus.cc:729] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f39ee03c04f4aa28b8bd0e710ed5087, State: Initialized, Role: FOLLOWER
I20260924 10:34:53.235771 4449 raft_consensus.cc:729] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2cace2800243a28690e68c290de9cd, State: Initialized, Role: FOLLOWER
I20260924 10:34:53.235896 4449 consensus_queue.cc:260] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.235930 4444 consensus_queue.cc:260] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.236018 4443 tablet_bootstrap.cc:492] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: Bootstrap starting.
I20260924 10:34:53.236193 4449 ts_tablet_manager.cc:1428] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:53.236142 4447 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.236265 4447 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:53.236717 4444 ts_tablet_manager.cc:1428] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:53.236203 4451 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.236963 4451 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:53.237010 4443 tablet_bootstrap.cc:654] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:53.238612 4443 tablet_bootstrap.cc:492] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: No bootstrap required, opened a new log
I20260924 10:34:53.238719 4443 ts_tablet_manager.cc:1397] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 10:34:53.239151 4443 raft_consensus.cc:348] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.239267 4443 raft_consensus.cc:374] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:53.239305 4443 raft_consensus.cc:729] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11450cccd0224021967ddcb1ae30ecf7, State: Initialized, Role: FOLLOWER
I20260924 10:34:53.239429 4443 consensus_queue.cc:260] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.239707 4443 ts_tablet_manager.cc:1428] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:34:53.239682 4448 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.239809 4448 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:53.391203 4448 raft_consensus.cc:482] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:53.391362 4448 raft_consensus.cc:504] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.392607 4448 leader_election.cc:288] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129:35577), df2cace2800243a28690e68c290de9cd (127.2.193.130:35845)
I20260924 10:34:53.397452 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.393207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:51249 (local address 127.2.193.129:35577)
0924 10:34:53.393330 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.393336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.393412 (+ 76us) server_negotiation.cc:407] Connection header received
0924 10:34:53.393646 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.393651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.393721 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.393817 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:53.394152 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.394929 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.396368 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.396774 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.396909 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.396969 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.397249 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.397334 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:34:53.397480 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.392871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35577 (local address 127.2.193.131:51249)
0924 10:34:53.393315 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.393329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.393490 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.393871 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.393873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.393893 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:53.394028 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.394035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.395045 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.395048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.396244 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.396254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.396907 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.396910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.397029 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.397105 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.397204 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.397302 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":155,"thread_start_us":74,"threads_started":1}
I20260924 10:34:53.397543 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.392944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:43085 (local address 127.2.193.130:35845)
0924 10:34:53.393188 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.393193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.393378 (+ 185us) server_negotiation.cc:407] Connection header received
0924 10:34:53.393543 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.393546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.393632 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.393719 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:53.394020 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.394599 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.396255 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.396898 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.397016 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.397064 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.397331 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.397407 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":62,"threads_started":1}
I20260924 10:34:53.397881 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.392871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:35845 (local address 127.2.193.131:43085)
0924 10:34:53.393284 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.393299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.393416 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.393743 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.393745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.393762 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:53.393888 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.393895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.394743 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.394747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.396015 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.396027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.397038 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.397041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.397197 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.397222 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.397665 (+ 443us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.397750 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260924 10:34:53.398087 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a34956e2554fec9087a911f38978fd" candidate_uuid: "11450cccd0224021967ddcb1ae30ecf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" is_pre_election: true
I20260924 10:34:53.398231 4094 raft_consensus.cc:2454] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11450cccd0224021967ddcb1ae30ecf7 in term 0.
I20260924 10:34:53.398360 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a34956e2554fec9087a911f38978fd" candidate_uuid: "11450cccd0224021967ddcb1ae30ecf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2cace2800243a28690e68c290de9cd" is_pre_election: true
I20260924 10:34:53.398497 4234 raft_consensus.cc:2454] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11450cccd0224021967ddcb1ae30ecf7 in term 0.
I20260924 10:34:53.398638 4308 leader_election.cc:302] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11450cccd0224021967ddcb1ae30ecf7, 9f39ee03c04f4aa28b8bd0e710ed5087; no voters:
I20260924 10:34:53.398895 4448 raft_consensus.cc:2781] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:53.398970 4448 raft_consensus.cc:482] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:53.399020 4448 raft_consensus.cc:3037] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:53.400056 4448 raft_consensus.cc:504] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.400440 4448 leader_election.cc:288] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [CANDIDATE]: Term 1 election: Requested vote from peers 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129:35577), df2cace2800243a28690e68c290de9cd (127.2.193.130:35845)
I20260924 10:34:53.400805 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a34956e2554fec9087a911f38978fd" candidate_uuid: "11450cccd0224021967ddcb1ae30ecf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2cace2800243a28690e68c290de9cd"
I20260924 10:34:53.400805 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a34956e2554fec9087a911f38978fd" candidate_uuid: "11450cccd0224021967ddcb1ae30ecf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087"
I20260924 10:34:53.400936 4094 raft_consensus.cc:3037] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:53.400954 4234 raft_consensus.cc:3037] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:53.402109 4234 raft_consensus.cc:2454] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11450cccd0224021967ddcb1ae30ecf7 in term 1.
I20260924 10:34:53.402109 4094 raft_consensus.cc:2454] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11450cccd0224021967ddcb1ae30ecf7 in term 1.
I20260924 10:34:53.402408 4305 leader_election.cc:302] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11450cccd0224021967ddcb1ae30ecf7, df2cace2800243a28690e68c290de9cd; no voters:
I20260924 10:34:53.402599 4448 raft_consensus.cc:2781] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:53.402778 4448 raft_consensus.cc:686] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [term 1 LEADER]: Becoming Leader. State: Replica: 11450cccd0224021967ddcb1ae30ecf7, State: Running, Role: LEADER
I20260924 10:34:53.403041 4448 consensus_queue.cc:237] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } }
I20260924 10:34:53.404120 4489 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: TxnStatusTablet state changed. Reason: New leader 11450cccd0224021967ddcb1ae30ecf7. Latest consensus state: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.404232 4489 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:34:53.404759 4491 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:34:53.404937 4491 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:34:53.405530 3935 catalog_manager.cc:5495] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131). New cstate: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } health_report { overall_health: HEALTHY } } }
I20260924 10:34:53.432544 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.428120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.2.193.131:44629)
0924 10:34:53.428255 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.428261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.428277 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:34:53.428580 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.428583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.428642 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.428766 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:53.429059 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.429682 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.430940 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.431157 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.431241 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.431648 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:53.432130 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:53.432241 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.432354 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.432411 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:34:53.432582 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.428033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:44629 (local address 127.0.0.1:57350)
0924 10:34:53.428206 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.428223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.428395 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.428802 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.428806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.428815 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:53.428937 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.428944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.429799 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.429803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.430807 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.430818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.431278 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.431282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.431383 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.431451 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:53.432239 (+ 788us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:53.432258 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.432334 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.432433 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:34:53.436025 4234 raft_consensus.cc:1261] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Refusing update from remote peer 11450cccd0224021967ddcb1ae30ecf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:53.436383 4094 raft_consensus.cc:1261] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Refusing update from remote peer 11450cccd0224021967ddcb1ae30ecf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:53.436580 4448 consensus_queue.cc:1035] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:53.436949 4489 consensus_queue.cc:1035] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:53.438555 4447 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: TxnStatusTablet state changed. Reason: New leader 11450cccd0224021967ddcb1ae30ecf7. Latest consensus state: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.438567 4451 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: TxnStatusTablet state changed. Reason: New leader 11450cccd0224021967ddcb1ae30ecf7. Latest consensus state: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.438683 4451 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:53.438684 4447 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:53.440863 4489 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.441006 4489 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:34:53.441665 4447 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.441792 4447 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P 9f39ee03c04f4aa28b8bd0e710ed5087: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:53.441766 4448 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.441869 4448 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P 11450cccd0224021967ddcb1ae30ecf7: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:34:53.442917 4451 tablet_replica.cc:439] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11450cccd0224021967ddcb1ae30ecf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 } } peers { permanent_uuid: "df2cace2800243a28690e68c290de9cd" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 35845 } } peers { permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 } } }
I20260924 10:34:53.443035 4451 tablet_replica.cc:442] T a6a34956e2554fec9087a911f38978fd P df2cace2800243a28690e68c290de9cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:53.452867 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.449053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.2.193.130:35845)
0924 10:34:53.449166 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.449170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.449333 (+ 163us) server_negotiation.cc:407] Connection header received
0924 10:34:53.449546 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.449548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.449603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.449718 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:53.449963 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.450487 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.451362 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.451591 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.451705 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.451994 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:53.452460 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:53.452591 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.452653 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.452711 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:34:53.464509 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.460789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:36159 (local address 127.2.193.131:44629)
0924 10:34:53.460922 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.460927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.461101 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:34:53.461242 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.461245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.461300 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.461377 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:53.461680 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.462448 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.463337 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.463684 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.463841 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.464081 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:53.464168 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:53.464241 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.464317 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.464369 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:34:53.464811 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.460669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:44629 (local address 127.2.193.130:36159)
0924 10:34:53.461008 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.461021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.461116 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.461440 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.461443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.461451 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:53.461558 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.461564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.462598 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.462601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.463209 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.463218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.463833 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.463835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.463896 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.463944 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:53.464247 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:53.464252 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.464300 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.464366 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":125,"threads_started":1}
I20260924 10:34:53.472152 4374 raft_consensus.cc:1261] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Refusing update from remote peer df2cace2800243a28690e68c290de9cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:53.472190 4094 raft_consensus.cc:1261] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Refusing update from remote peer df2cace2800243a28690e68c290de9cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:53.472687 4474 consensus_queue.cc:1035] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11450cccd0224021967ddcb1ae30ecf7" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:53.472935 4451 consensus_queue.cc:1035] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f39ee03c04f4aa28b8bd0e710ed5087" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:53.478757 4496 mvcc.cc:205] Tried to move back new op lower bound from 7332847998857437184 to 7332847997452296192. Current Snapshot: MvccSnapshot[applied={T|T < 7332847998857437184}]
I20260924 10:34:53.497239 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.493761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:53161 (local address 127.2.193.130:35845)
0924 10:34:53.493886 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.493892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.493905 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:34:53.493949 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.493952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.494006 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.494087 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:53.494355 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.494945 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.496076 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.496433 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.496535 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.496808 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:53.496890 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:53.496960 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.497015 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.497069 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:34:53.497273 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.493408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:35845 (local address 127.2.193.131:53161)
0924 10:34:53.493716 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:53.493734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:53.493834 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:53.494096 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:53.494099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:53.494106 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:53.494249 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.494254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.495055 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.495059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.495952 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:53.495961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.496534 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.496538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:53.496618 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.496667 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:53.496957 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:53.496960 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:34:53.497003 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 10:34:53.497068 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260924 10:34:53.526202 3936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:53.526397 3936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:53.529109 3936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b1c6bddf05445ee9fca46eb2ca6d03f: Sending DeleteTablet for 3 replicas of tablet 344f3f29349d40bbb7139c9ada2cc8b1
I20260924 10:34:53.529928 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 344f3f29349d40bbb7139c9ada2cc8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:53 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:54758
I20260924 10:34:53.530342 4515 tablet_replica.cc:330] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: stopping tablet replica
I20260924 10:34:53.530493 4515 raft_consensus.cc:2229] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:53.530516 4214 tablet_service.cc:1514] Processing DeleteTablet for tablet 344f3f29349d40bbb7139c9ada2cc8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:53 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:38412
I20260924 10:34:53.530618 4515 raft_consensus.cc:2258] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:53.530865 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 344f3f29349d40bbb7139c9ada2cc8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:53 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:57348
I20260924 10:34:53.531361 4515 ts_tablet_manager.cc:1905] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:53.532598 4516 tablet_replica.cc:330] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: stopping tablet replica
I20260924 10:34:53.532789 4516 raft_consensus.cc:2229] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:34:53.533244 4516 raft_consensus.cc:2258] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:53.534018 4516 ts_tablet_manager.cc:1905] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:53.534888 4515 ts_tablet_manager.cc:1918] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:53.534981 4515 log.cc:1196] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/344f3f29349d40bbb7139c9ada2cc8b1
I20260924 10:34:53.535246 4515 ts_tablet_manager.cc:1939] T 344f3f29349d40bbb7139c9ada2cc8b1 P 9f39ee03c04f4aa28b8bd0e710ed5087: Deleting consensus metadata
I20260924 10:34:53.536055 4517 tablet_replica.cc:330] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: stopping tablet replica
I20260924 10:34:53.536024 3924 catalog_manager.cc:4841] TS 9f39ee03c04f4aa28b8bd0e710ed5087 (127.2.193.129:35577): tablet 344f3f29349d40bbb7139c9ada2cc8b1 (table test-table [id=d3e5ed0925834d72b56331d558482781]) successfully deleted
I20260924 10:34:53.536187 4517 raft_consensus.cc:2229] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:53.536306 4517 raft_consensus.cc:2258] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:53.537025 4517 ts_tablet_manager.cc:1905] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:53.538116 4516 ts_tablet_manager.cc:1918] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:53.538252 4516 log.cc:1196] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/344f3f29349d40bbb7139c9ada2cc8b1
I20260924 10:34:53.538605 4516 ts_tablet_manager.cc:1939] T 344f3f29349d40bbb7139c9ada2cc8b1 P df2cace2800243a28690e68c290de9cd: Deleting consensus metadata
I20260924 10:34:53.539419 3921 catalog_manager.cc:4841] TS df2cace2800243a28690e68c290de9cd (127.2.193.130:35845): tablet 344f3f29349d40bbb7139c9ada2cc8b1 (table test-table [id=d3e5ed0925834d72b56331d558482781]) successfully deleted
I20260924 10:34:53.540551 4517 ts_tablet_manager.cc:1918] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:53.540637 4517 log.cc:1196] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/344f3f29349d40bbb7139c9ada2cc8b1
I20260924 10:34:53.540912 4517 ts_tablet_manager.cc:1939] T 344f3f29349d40bbb7139c9ada2cc8b1 P 11450cccd0224021967ddcb1ae30ecf7: Deleting consensus metadata
I20260924 10:34:53.541635 3924 catalog_manager.cc:4841] TS 11450cccd0224021967ddcb1ae30ecf7 (127.2.193.131:44629): tablet 344f3f29349d40bbb7139c9ada2cc8b1 (table test-table [id=d3e5ed0925834d72b56331d558482781]) successfully deleted
Sep 24 10:34:53 dist-test-slave-8wc7 krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:34:53.554562 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.541252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.2.193.129:35577)
0924 10:34:53.541384 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.541389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.542110 (+ 721us) server_negotiation.cc:407] Connection header received
0924 10:34:53.542256 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.542260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.542312 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.542399 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:53.543362 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.544166 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.545397 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.545754 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.545864 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.549960 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:53.549976 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:53.549993 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:53.550026 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:53.552221 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:53.552794 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:53.552801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:53.552803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:53.552904 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:53.553304 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:53.553307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:53.553309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:53.553656 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:53.553863 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.554194 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.554331 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:34:53.555104 4134 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54778
I20260924 10:34:53.566632 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:53.557517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.2.193.190:42369)
0924 10:34:53.557656 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:53.557662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:53.557681 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:34:53.557741 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:53.557746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:53.557795 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:53.557897 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:53.558722 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.559560 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.560742 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:53.561103 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:53.561222 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:53.563068 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:53.563072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:53.563075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:53.563106 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:53.564810 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:53.565306 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:53.565309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:53.565312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:53.565404 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:53.565697 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:53.565700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:53.565702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:53.565929 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:53.566061 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:53.566304 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 10:34:53.566455 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 10:34:53.567152 3936 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54132
I20260924 10:34:53.568854 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 4009
I20260924 10:34:53.580029 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 4149
I20260924 10:34:53.585722 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 4289
I20260924 10:34:53.592278 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 3905
2026-09-24T10:34:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3081 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:34:53 dist-test-slave-8wc7 krb5kdc[4524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:34:53 dist-test-slave-8wc7 krb5kdc[4524](info): set up 2 sockets
Sep 24 10:34:53 dist-test-slave-8wc7 krb5kdc[4524](info): commencing operation
krb5kdc: starting...
W20260924 10:34:54.625102 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:34:54 dist-test-slave-8wc7 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:34:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:34:54.746408 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36467
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:36467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:54.856884 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:54.857177 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:54.857266 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:54.861152 4537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:34:54.861292 4537 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:34:54.861325 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:54.861366 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:54.861415 4537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:34:54.861454 4537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:34:54.866505 4537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:36467
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:54.867512 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:54.868481 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:54.876266 4545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:54.876274 4542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:54.876338 4543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:54.876700 4537 server_base.cc:1034] running on GCE node
I20260924 10:34:54.877233 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:54.878199 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:54.879379 4537 hybrid_clock.cc:648] HybridClock initialized: now 1790246094879368 us; error 23 us; skew 500 ppm
Sep 24 10:34:54 dist-test-slave-8wc7 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:54.881981 4537 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:34:54.882717 4537 webserver.cc:460] Webserver started at http://127.2.193.190:43439/ using document root <none> and password file <none>
I20260924 10:34:54.883180 4537 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:54.883234 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:54.883380 4537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:54.885260 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "dfefde44165a4bc0ae9b317cc97d3f8c"
format_stamp: "Formatted at 2026-09-24 10:34:54 on dist-test-slave-8wc7"
server_key: "f41355031d207ef0654011c05f800134"
server_key_iv: "95731399b0cb4846e3c67ee2b92d9151"
server_key_version: "encryptionkey@0"
I20260924 10:34:54.885771 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "dfefde44165a4bc0ae9b317cc97d3f8c"
format_stamp: "Formatted at 2026-09-24 10:34:54 on dist-test-slave-8wc7"
server_key: "f41355031d207ef0654011c05f800134"
server_key_iv: "95731399b0cb4846e3c67ee2b92d9151"
server_key_version: "encryptionkey@0"
I20260924 10:34:54.889115 4537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:34:54.891351 4552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:54.892459 4537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:34:54.892584 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "dfefde44165a4bc0ae9b317cc97d3f8c"
format_stamp: "Formatted at 2026-09-24 10:34:54 on dist-test-slave-8wc7"
server_key: "f41355031d207ef0654011c05f800134"
server_key_iv: "95731399b0cb4846e3c67ee2b92d9151"
server_key_version: "encryptionkey@0"
I20260924 10:34:54.892675 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:54.903987 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:54.906526 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:54.906708 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:54.913801 4537 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:36467
I20260924 10:34:54.913887 4614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:36467 every 8 connection(s)
I20260924 10:34:54.914687 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:34:54.917351 4615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:54.922544 4615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Bootstrap starting.
I20260924 10:34:54.922552 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 4537
I20260924 10:34:54.922668 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:34:54.922911 2822 external_mini_cluster.cc:1428] Setting key de397f29370a54da4f6a3bea75aa2b1e
I20260924 10:34:54.925001 4615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:54.925772 4615 log.cc:824] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:54.927534 4615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: No bootstrap required, opened a new log
Sep 24 10:34:54 dist-test-slave-8wc7 krb5kdc[4524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:54.931039 4615 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36467 } }
I20260924 10:34:54.931306 4615 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:54.931394 4615 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfefde44165a4bc0ae9b317cc97d3f8c, State: Initialized, Role: FOLLOWER
I20260924 10:34:54.931910 4615 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36467 } }
I20260924 10:34:54.932085 4615 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:34:54.932161 4615 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:34:54.932267 4615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:54.933264 4615 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36467 } }
I20260924 10:34:54.933647 4615 leader_election.cc:302] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfefde44165a4bc0ae9b317cc97d3f8c; no voters:
I20260924 10:34:54.934013 4615 leader_election.cc:288] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:34:54.934099 4620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:54.934338 4620 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [term 1 LEADER]: Becoming Leader. State: Replica: dfefde44165a4bc0ae9b317cc97d3f8c, State: Running, Role: LEADER
I20260924 10:34:54.934716 4620 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36467 } }
I20260924 10:34:54.935385 4615 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:34:54.936324 4622 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfefde44165a4bc0ae9b317cc97d3f8c. Latest consensus state: current_term: 1 leader_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36467 } } }
I20260924 10:34:54.936447 4622 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:54.936997 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:54.924170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.2.193.190:36467)
0924 10:34:54.924582 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:54.924596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:54.924628 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:34:54.925325 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:54.925344 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:54.925643 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:54.925959 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:54.926704 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:54.927682 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:54.928523 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:54.928976 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:54.929147 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:54.931554 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:54.931559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:54.931571 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:54.931596 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:54.933379 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:54.933924 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:54.933928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:54.933932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:54.934088 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:54.934436 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:54.934440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:54.934441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:54.934923 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:54.935124 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:54.935410 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:34:54.935863 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":113,"threads_started":1}
I20260924 10:34:54.937004 4621 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfefde44165a4bc0ae9b317cc97d3f8c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36467 } } }
I20260924 10:34:54.938197 4621 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c [sys.catalog]: This master's current role is: LEADER
I20260924 10:34:54.939101 4629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:34:54.941690 4629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:34:54.946887 4629 catalog_manager.cc:1347] Generated new cluster ID: 4847e263b0f14f9fb3906dd7bde1cbc6
I20260924 10:34:54.946952 4629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:34:54.953336 4629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:34:54.954505 4629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:34:54.963124 4629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Generated new TSK 0
I20260924 10:34:54.963883 4629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:34:54.998106 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36467
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:55.144312 4641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:55.144582 4641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:55.144665 4641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:55.148676 4641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:55.148738 4641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:55.148778 4641 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:34:55.148792 4641 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:34:55.148864 4641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:34:55.154361 4641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:55.155529 4641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:55.156548 4641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:55.166363 4646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:55.166496 4641 server_base.cc:1034] running on GCE node
W20260924 10:34:55.166356 4649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:55.166662 4647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:55.167078 4641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:55.167737 4641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:55.168919 4641 hybrid_clock.cc:648] HybridClock initialized: now 1790246095168902 us; error 28 us; skew 500 ppm
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:55.171608 4641 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:34:55.172441 4641 webserver.cc:460] Webserver started at http://127.2.193.129:35429/ using document root <none> and password file <none>
I20260924 10:34:55.172998 4641 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:55.173084 4641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:55.173357 4641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:55.175419 4641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "4f2632bf24e348278c70983adbc6b703"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "025bd9f0428e95066d244176bf555c45"
server_key_iv: "edd06bd35e15b3ccad9919c9281f3e68"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.175992 4641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "4f2632bf24e348278c70983adbc6b703"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "025bd9f0428e95066d244176bf555c45"
server_key_iv: "edd06bd35e15b3ccad9919c9281f3e68"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.179687 4641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:34:55.182801 4656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:55.184026 4641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:34:55.184165 4641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "4f2632bf24e348278c70983adbc6b703"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "025bd9f0428e95066d244176bf555c45"
server_key_iv: "edd06bd35e15b3ccad9919c9281f3e68"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.184290 4641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:55.202852 4641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:55.206102 4641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:55.206316 4641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:55.207777 4641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:55.207854 4641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:55.207929 4641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:55.207975 4641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:55.219304 4641 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:34679
I20260924 10:34:55.219352 4771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:34679 every 8 connection(s)
I20260924 10:34:55.220484 4641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:34:55.222388 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.208674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:38885 (local address 127.2.193.190:36467)
0924 10:34:55.208803 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.208807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.209901 (+ 1094us) server_negotiation.cc:407] Connection header received
0924 10:34:55.210925 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.210940 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.211010 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.211115 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.212300 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.212939 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.214204 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.214435 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.214553 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.218424 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.218428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.218431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.218461 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.220245 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.220813 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.220820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.220824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.220925 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.221416 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.221419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.221420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.221605 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.221682 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.222151 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.222268 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:34:55.223241 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.209281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36467 (local address 127.2.193.129:38885)
0924 10:34:55.209752 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.209794 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.210677 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.211318 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.211330 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.211681 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:55.212148 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.212161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.213061 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.213064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.214075 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.214085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.214593 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.214597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.214823 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.215534 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:55.215564 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:55.218244 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:55.220370 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.220374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.220385 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.220696 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.221071 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.221074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.221076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.221301 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.221705 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:55.221710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:55.221942 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.222092 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.222346 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260924 10:34:55.224531 4772 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36467
I20260924 10:34:55.224665 4772 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:55.225255 4772 heartbeater.cc:511] Master 127.2.193.190:36467 requested a full tablet report, sending...
I20260924 10:34:55.225309 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 4641
I20260924 10:34:55.225445 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:34:55.225782 2822 external_mini_cluster.cc:1428] Setting key 2871f3da68a4bf2c470e6b5c957f766f
I20260924 10:34:55.227007 4568 ts_manager.cc:194] Registered new tserver with Master: 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679)
I20260924 10:34:55.228171 4568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:38885
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:34:55.236878 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.226959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:45045 (local address 127.2.193.190:36467)
0924 10:34:55.227074 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.227080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.229040 (+ 1960us) server_negotiation.cc:407] Connection header received
0924 10:34:55.229245 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.229248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.229298 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.229380 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.229816 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.230664 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.231611 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.232033 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.232166 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.233425 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.233432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.233436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.233472 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.235117 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.235600 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.235604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.235606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.235692 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.236075 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.236081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.236086 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.236276 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.236355 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.236606 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.236699 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:34:55.236884 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.228675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36467 (local address 127.2.193.129:45045)
0924 10:34:55.228959 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.228976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.229110 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.229395 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.229398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.229610 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:55.229702 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.229708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.230796 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.230799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.231475 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.231488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.232166 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.232170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.232257 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.232869 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:55.232881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:55.233283 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:55.235278 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.235281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.235282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.235499 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.235865 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.235871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.235876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.235974 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.236367 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:55.236369 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:55.236456 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.236550 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.236665 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:34:55.256132 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36467
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:55.368620 4781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:55.368934 4781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:55.369016 4781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:55.373200 4781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:55.373289 4781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:55.373361 4781 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:34:55.373409 4781 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:34:55.373485 4781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:34:55.379083 4781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:55.380259 4781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:55.381378 4781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:55.391448 4787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:55.391448 4786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:55.391445 4789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:55.391724 4781 server_base.cc:1034] running on GCE node
I20260924 10:34:55.392230 4781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:55.392904 4781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:55.394093 4781 hybrid_clock.cc:648] HybridClock initialized: now 1790246095394059 us; error 51 us; skew 500 ppm
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:55.396944 4781 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:34:55.397876 4781 webserver.cc:460] Webserver started at http://127.2.193.130:44097/ using document root <none> and password file <none>
I20260924 10:34:55.398409 4781 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:55.398478 4781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:55.398648 4781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:55.400709 4781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "b9355d899d4e4970ad331e6e09ee4411"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "2ee19fc674884550f44382c8ee5a897e"
server_key_iv: "9742c797673e9bb89c8092b86e134521"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.401304 4781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "b9355d899d4e4970ad331e6e09ee4411"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "2ee19fc674884550f44382c8ee5a897e"
server_key_iv: "9742c797673e9bb89c8092b86e134521"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.405301 4781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:34:55.407686 4796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:55.408942 4781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:55.409051 4781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "b9355d899d4e4970ad331e6e09ee4411"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "2ee19fc674884550f44382c8ee5a897e"
server_key_iv: "9742c797673e9bb89c8092b86e134521"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.409144 4781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:55.426610 4781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:55.430107 4781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:55.430272 4781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:55.431744 4781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:55.431825 4781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:55.431902 4781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:55.431947 4781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:55.442783 4781 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:36661
I20260924 10:34:55.442853 4911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:36661 every 8 connection(s)
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:55.443902 4781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:34:55.447293 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.432601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:39419 (local address 127.2.193.190:36467)
0924 10:34:55.432758 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.432764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.433758 (+ 994us) server_negotiation.cc:407] Connection header received
0924 10:34:55.434936 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.434939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.435016 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.435126 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.437456 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.438387 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.439961 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.440360 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.440484 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.443701 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.443705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.443707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.443734 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.445434 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.445961 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.445964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.445965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.446049 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.446355 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.446357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.446358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.446532 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.446603 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.447063 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.447162 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:34:55.448349 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.433070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36467 (local address 127.2.193.130:39419)
0924 10:34:55.433575 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.433628 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.434690 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.435397 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.435407 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.436403 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:55.437286 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.437310 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.438565 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.438571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.439815 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.439825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.440569 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.440573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.440788 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.441603 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:55.441630 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:55.443501 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:55.445554 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.445566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.445578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.445845 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.446127 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.446130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.446131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.446270 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.446614 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:55.446619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:55.446848 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.447006 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.447419 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260924 10:34:55.449615 4912 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36467
I20260924 10:34:55.449779 4912 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:55.450325 4912 heartbeater.cc:511] Master 127.2.193.190:36467 requested a full tablet report, sending...
I20260924 10:34:55.451330 4569 ts_manager.cc:194] Registered new tserver with Master: b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661)
I20260924 10:34:55.452119 4569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:39419
I20260924 10:34:55.452491 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 4781
I20260924 10:34:55.452616 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:34:55.452940 2822 external_mini_cluster.cc:1428] Setting key 04cbb5ec5ea26f7ade69a8e2c470a354
I20260924 10:34:55.459520 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.451696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36467 (local address 127.2.193.130:57665)
0924 10:34:55.451989 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.452004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.452102 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.452392 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.452395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.452668 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:55.452816 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.452825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.453729 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.453732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.454528 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.454538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.455021 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.455024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.455118 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.455676 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:55.455693 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:55.456185 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:55.457966 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.457971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.457974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.458174 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.458461 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.458466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.458468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.458554 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.458970 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:55.458973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:55.459074 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.459180 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.459337 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260924 10:34:55.459510 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.451748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:57665 (local address 127.2.193.190:36467)
0924 10:34:55.451881 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.451886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.452076 (+ 190us) server_negotiation.cc:407] Connection header received
0924 10:34:55.452187 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.452189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.452276 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.452376 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.452950 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.453617 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.454662 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.454910 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.455018 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.456335 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.456338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.456340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.456361 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.457847 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.458274 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.458279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.458281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.458363 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.458643 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.458647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.458649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.458864 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.458983 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.459232 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.459338 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:34:55.480700 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36467
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:34:55.595465 4921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:34:55.595755 4921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:34:55.595817 4921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:34:55.599992 4921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:34:55.600064 4921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:34:55.600109 4921 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:34:55.600124 4921 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:34:55.600157 4921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:34:55.605840 4921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:34:55.606997 4921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:34:55.608043 4921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:34:55.618408 4929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:34:55.618387 4926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:55.618561 4921 server_base.cc:1034] running on GCE node
W20260924 10:34:55.618752 4927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:34:55.619172 4921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:34:55.619861 4921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:34:55.621049 4921 hybrid_clock.cc:648] HybridClock initialized: now 1790246095621033 us; error 38 us; skew 500 ppm
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:34:55.623755 4921 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:34:55.624751 4921 webserver.cc:460] Webserver started at http://127.2.193.131:40377/ using document root <none> and password file <none>
I20260924 10:34:55.625468 4921 fs_manager.cc:359] Metadata directory not provided
I20260924 10:34:55.625561 4921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:34:55.625773 4921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:34:55.627990 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "92752c45b5ab441d9bf7db71c4bc425e"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "f6ea4e29a212336505baf1f8b75970a3"
server_key_iv: "f8d0953ac00c233ceb0352ff818d5129"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.628571 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "92752c45b5ab441d9bf7db71c4bc425e"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "f6ea4e29a212336505baf1f8b75970a3"
server_key_iv: "f8d0953ac00c233ceb0352ff818d5129"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.632397 4921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:34:55.635082 4936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:55.636449 4921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:34:55.636590 4921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "92752c45b5ab441d9bf7db71c4bc425e"
format_stamp: "Formatted at 2026-09-24 10:34:55 on dist-test-slave-8wc7"
server_key: "f6ea4e29a212336505baf1f8b75970a3"
server_key_iv: "f8d0953ac00c233ceb0352ff818d5129"
server_key_version: "encryptionkey@0"
I20260924 10:34:55.636713 4921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:34:55.666095 4921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:34:55.669277 4921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:34:55.669503 4921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:34:55.671069 4921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:34:55.671147 4921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:34:55.671221 4921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:34:55.671265 4921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:55.682227 4921 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:41725
I20260924 10:34:55.682485 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:41725 every 8 connection(s)
I20260924 10:34:55.683418 4921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:34:55.686134 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.671877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:35387 (local address 127.2.193.190:36467)
0924 10:34:55.672032 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.672037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.672982 (+ 945us) server_negotiation.cc:407] Connection header received
0924 10:34:55.674389 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.674393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.674452 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.674560 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.675952 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.676491 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.677898 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.678204 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.678341 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.682376 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.682382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.682387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.682420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.683919 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.684524 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.684528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.684530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.684620 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.684968 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.684971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.684974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.685246 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.685346 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.685850 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.685977 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:34:55.686937 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.672340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36467 (local address 127.2.193.131:35387)
0924 10:34:55.672795 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.672847 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.674118 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.674741 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.674748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.675203 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:55.675801 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.675815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.676620 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.676623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.677763 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.677773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.678403 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.678409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.678677 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.680101 (+ 1424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:55.680127 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:55.682133 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:55.684054 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.684059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.684074 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.684396 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.684700 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.684705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.684708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.684858 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.685352 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:55.685361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:55.685610 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.685796 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.686131 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260924 10:34:55.688340 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 4921
I20260924 10:34:55.688462 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:34:55.688476 5052 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36467
I20260924 10:34:55.688647 5052 heartbeater.cc:464] Registering TS with master...
I20260924 10:34:55.688789 2822 external_mini_cluster.cc:1428] Setting key dcc064038838194f2f90dbd29d735a89
I20260924 10:34:55.689301 5052 heartbeater.cc:511] Master 127.2.193.190:36467 requested a full tablet report, sending...
I20260924 10:34:55.690371 4569 ts_manager.cc:194] Registered new tserver with Master: 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725)
I20260924 10:34:55.691108 4569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:35387
I20260924 10:34:55.697470 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.690705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:45329 (local address 127.2.193.190:36467)
0924 10:34:55.690815 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.690818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.691087 (+ 269us) server_negotiation.cc:407] Connection header received
0924 10:34:55.691254 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.691256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.691299 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.691383 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.691876 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.692360 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.693410 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.693618 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.693695 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.694688 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.694690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.694693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.694714 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.696001 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.696364 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.696366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.696367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.696435 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.696696 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.696698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.696699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.696858 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.696934 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.697270 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.697362 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:34:55.697479 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.690647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36467 (local address 127.2.193.131:45329)
0924 10:34:55.691002 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.691013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.691127 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.691414 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.691417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.691640 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:55.691763 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.691770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.692493 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.692496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.693312 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.693321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.693716 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.693719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.693783 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.694241 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:55.694252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:55.694569 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:55.696091 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.696093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.696095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.696284 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.696526 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.696528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.696529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.696600 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.696946 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:55.696948 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:55.697043 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.697158 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.697339 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":81,"threads_started":1}
I20260924 10:34:55.702616 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:55.728387 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.719951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.2.193.190:36467)
0924 10:34:55.720087 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.720091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.720102 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:34:55.720145 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.720148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.720186 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.720283 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.721021 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.721629 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.722474 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.722696 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.722806 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.725400 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.725403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.725406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.725433 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.726666 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.727113 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.727117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.727119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.727206 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.727515 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.727519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.727521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.727711 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.727853 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.728089 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.728212 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:34:55.731510 4569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:34:55.734599 4569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:55.750280 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.744061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:34679 (local address 127.0.0.1:48064)
0924 10:34:55.744480 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.744529 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.744675 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.745565 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.745571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.745599 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:55.745757 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.745765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.747186 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.747189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.748451 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.748466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.749624 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.749627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.749746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.749784 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.749925 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.750127 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":4,"thread_start_us":94,"threads_started":1}
I20260924 10:34:55.750286 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.744181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.2.193.129:34679)
0924 10:34:55.744542 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.744549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.744821 (+ 272us) server_negotiation.cc:407] Connection header received
0924 10:34:55.744914 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.744922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.745147 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.745354 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:55.745900 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.747068 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.748584 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.749497 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.749623 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.749785 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.749954 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.750126 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":73,"threads_started":1}
I20260924 10:34:55.750635 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.744120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36661 (local address 127.0.0.1:59030)
0924 10:34:55.745263 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.745279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.745354 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.746487 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.746490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.746509 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:55.746585 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.746590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.747855 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.747858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.749104 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.749114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.750257 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.750261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.750341 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.750361 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.750412 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.750502 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":46,"thread_start_us":95,"threads_started":1}
I20260924 10:34:55.751235 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.744954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.2.193.130:36661)
0924 10:34:55.746020 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.746031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.746053 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:34:55.746121 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.746126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.746303 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.746482 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:55.746699 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.747724 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.749248 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.750130 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.750822 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.750960 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.751010 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.751096 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":108,"threads_started":1}
I20260924 10:34:55.751888 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.745337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.2.193.131:41725)
0924 10:34:55.745664 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.745675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.745756 (+ 81us) server_negotiation.cc:407] Connection header received
0924 10:34:55.745852 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.745860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.746027 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.746222 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:55.746405 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.747309 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.748861 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.749696 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.749799 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.749929 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.751647 (+ 1718us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.751748 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":117,"threads_started":1}
I20260924 10:34:55.752137 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.744814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41725 (local address 127.0.0.1:44618)
0924 10:34:55.745666 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.745677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.745759 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.746193 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.746196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.746213 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:55.746311 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.746318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.747446 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.747450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.748728 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.748740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.750003 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.750007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.750131 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.750156 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.750228 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.750301 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":27,"thread_start_us":96,"threads_started":1}
I20260924 10:34:55.753615 4706 tablet_service.cc:1467] Processing CreateTablet for tablet 1338bbb1ea5b49098e48370787f9513d (DEFAULT_TABLE table=test-table [id=6ec4dd0cdcb545c0a56c6cf20c9b93d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:55.753906 4846 tablet_service.cc:1467] Processing CreateTablet for tablet 1338bbb1ea5b49098e48370787f9513d (DEFAULT_TABLE table=test-table [id=6ec4dd0cdcb545c0a56c6cf20c9b93d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:55.754453 4986 tablet_service.cc:1467] Processing CreateTablet for tablet 1338bbb1ea5b49098e48370787f9513d (DEFAULT_TABLE table=test-table [id=6ec4dd0cdcb545c0a56c6cf20c9b93d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:55.754801 4706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338bbb1ea5b49098e48370787f9513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:55.754801 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338bbb1ea5b49098e48370787f9513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:55.755234 4986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1338bbb1ea5b49098e48370787f9513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:55.761869 5075 tablet_bootstrap.cc:492] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Bootstrap starting.
I20260924 10:34:55.763187 5076 tablet_bootstrap.cc:492] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Bootstrap starting.
I20260924 10:34:55.764334 5075 tablet_bootstrap.cc:654] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:55.764393 5077 tablet_bootstrap.cc:492] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Bootstrap starting.
I20260924 10:34:55.765256 5075 log.cc:824] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:55.765405 5076 tablet_bootstrap.cc:654] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:55.766168 5076 log.cc:824] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:55.766748 5077 tablet_bootstrap.cc:654] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:55.767565 5075 tablet_bootstrap.cc:492] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: No bootstrap required, opened a new log
I20260924 10:34:55.767571 5077 log.cc:824] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Log is configured to *not* fsync() on all Append() calls
I20260924 10:34:55.767766 5075 ts_tablet_manager.cc:1397] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:34:55.768045 5076 tablet_bootstrap.cc:492] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: No bootstrap required, opened a new log
I20260924 10:34:55.768218 5076 ts_tablet_manager.cc:1397] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:34:55.769375 5077 tablet_bootstrap.cc:492] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: No bootstrap required, opened a new log
I20260924 10:34:55.769575 5077 ts_tablet_manager.cc:1397] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:34:55.771891 5075 raft_consensus.cc:348] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.772186 5075 raft_consensus.cc:374] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:55.772244 5075 raft_consensus.cc:729] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92752c45b5ab441d9bf7db71c4bc425e, State: Initialized, Role: FOLLOWER
I20260924 10:34:55.772346 5076 raft_consensus.cc:348] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.772342 5077 raft_consensus.cc:348] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.772572 5076 raft_consensus.cc:374] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:55.772598 5077 raft_consensus.cc:374] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:55.772626 5076 raft_consensus.cc:729] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2632bf24e348278c70983adbc6b703, State: Initialized, Role: FOLLOWER
I20260924 10:34:55.772655 5077 raft_consensus.cc:729] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9355d899d4e4970ad331e6e09ee4411, State: Initialized, Role: FOLLOWER
I20260924 10:34:55.772773 5075 consensus_queue.cc:260] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.773061 5076 consensus_queue.cc:260] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.773077 5077 consensus_queue.cc:260] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.773892 5075 ts_tablet_manager.cc:1428] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:34:55.773892 5076 ts_tablet_manager.cc:1428] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:34:55.774049 5077 ts_tablet_manager.cc:1428] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 10:34:55.774143 5052 heartbeater.cc:503] Master 127.2.193.190:36467 was elected leader, sending a full tablet report...
I20260924 10:34:55.774230 4772 heartbeater.cc:503] Master 127.2.193.190:36467 was elected leader, sending a full tablet report...
I20260924 10:34:55.774327 4912 heartbeater.cc:503] Master 127.2.193.190:36467 was elected leader, sending a full tablet report...
I20260924 10:34:55.832573 5083 raft_consensus.cc:482] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:55.832832 5083 raft_consensus.cc:504] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.834028 5083 leader_election.cc:288] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725)
I20260924 10:34:55.838568 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.834301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:34679 (local address 127.2.193.130:45953)
0924 10:34:55.834455 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.834482 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.834592 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.834867 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.834870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.834906 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:55.835044 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.835051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.835895 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.835898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.837202 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.837215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.838170 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.838173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.838255 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.838275 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.838336 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.838418 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":21,"spinlock_wait_cycles":3456}
I20260924 10:34:55.838609 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.834484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:45953 (local address 127.2.193.129:34679)
0924 10:34:55.834607 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.834611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.834647 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:34:55.834708 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.834710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.834758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.834848 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:55.835184 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.835790 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.837395 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.838032 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.838153 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.838207 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.838372 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.838480 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:34:55.839293 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.834391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:57153 (local address 127.2.193.131:41725)
0924 10:34:55.834545 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.834551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.834925 (+ 374us) server_negotiation.cc:407] Connection header received
0924 10:34:55.835031 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.835034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.835083 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.835161 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:55.835421 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.835949 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.837937 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.838571 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.838688 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.838734 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.839137 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.839204 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:34:55.839305 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.834301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41725 (local address 127.2.193.130:57153)
0924 10:34:55.834830 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.834845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.834938 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.835184 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.835187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.835203 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:55.835315 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.835321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.836066 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.836069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.837823 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.837832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.838793 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.838796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.838932 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.838954 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.839096 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.839166 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":179,"thread_start_us":143,"threads_started":1}
I20260924 10:34:55.839344 4726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338bbb1ea5b49098e48370787f9513d" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703" is_pre_election: true
I20260924 10:34:55.839715 4726 raft_consensus.cc:2454] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 0.
I20260924 10:34:55.839910 5006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338bbb1ea5b49098e48370787f9513d" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92752c45b5ab441d9bf7db71c4bc425e" is_pre_election: true
I20260924 10:34:55.840184 5006 raft_consensus.cc:2454] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 0.
I20260924 10:34:55.840334 4798 leader_election.cc:302] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2632bf24e348278c70983adbc6b703, b9355d899d4e4970ad331e6e09ee4411; no voters:
I20260924 10:34:55.840641 5083 raft_consensus.cc:2781] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:55.840749 5083 raft_consensus.cc:482] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:55.840811 5083 raft_consensus.cc:3037] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:55.842015 5083 raft_consensus.cc:504] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.842371 5083 leader_election.cc:288] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 election: Requested vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725)
I20260924 10:34:55.842713 5006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338bbb1ea5b49098e48370787f9513d" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92752c45b5ab441d9bf7db71c4bc425e"
I20260924 10:34:55.842746 4726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1338bbb1ea5b49098e48370787f9513d" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703"
I20260924 10:34:55.842818 5006 raft_consensus.cc:3037] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:55.842909 4726 raft_consensus.cc:3037] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:55.843868 5006 raft_consensus.cc:2454] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 1.
I20260924 10:34:55.843952 4726 raft_consensus.cc:2454] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 1.
I20260924 10:34:55.844192 4800 leader_election.cc:302] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92752c45b5ab441d9bf7db71c4bc425e, b9355d899d4e4970ad331e6e09ee4411; no voters:
I20260924 10:34:55.844373 5083 raft_consensus.cc:2781] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:55.844656 5083 raft_consensus.cc:686] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 1 LEADER]: Becoming Leader. State: Replica: b9355d899d4e4970ad331e6e09ee4411, State: Running, Role: LEADER
I20260924 10:34:55.844990 5083 consensus_queue.cc:237] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.848279 4567 catalog_manager.cc:5495] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130). New cstate: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } health_report { overall_health: UNKNOWN } } }
Sep 24 10:34:55 dist-test-slave-8wc7 krb5kdc[4524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:55.903218 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.895025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.2.193.190:36467)
0924 10:34:55.895142 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:55.895146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:55.895429 (+ 283us) server_negotiation.cc:407] Connection header received
0924 10:34:55.895552 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:55.895554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:55.895595 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:55.895674 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:55.896396 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.896952 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.897863 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.898068 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.898146 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.900395 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:55.900398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:55.900400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:55.900420 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:55.901739 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.902172 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.902174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.902175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.902242 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:55.902467 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:55.902469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:55.902470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:55.902634 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:55.902704 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:55.902964 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 10:34:55.903096 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":43}
I20260924 10:34:55.903220 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:55.894976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36467 (local address 127.0.0.1:56590)
0924 10:34:55.895292 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:55.895310 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:55.895466 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:55.895709 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:55.895711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:55.896118 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:34:55.896286 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.896292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.897050 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.897052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.897760 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:55.897769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:55.898164 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:55.898166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:55.898235 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:55.898837 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:34:55.898848 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:34:55.900256 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:34:55.901835 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.901843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.901848 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.902089 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.902316 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:34:55.902318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:34:55.902320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:34:55.902392 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:34:55.902717 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:34:55.902722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:34:55.902821 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 10:34:55.902908 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:34:55.903041 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":149,"threads_started":1}
I20260924 10:34:55.907330 4567 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:56590:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:34:55.908080 4567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:55.913327 4706 tablet_service.cc:1467] Processing CreateTablet for tablet aecec804a1674d53b221d77b3a16ead1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2bd8d8380544a878e2300dd65a1bfec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:34:55.913582 4846 tablet_service.cc:1467] Processing CreateTablet for tablet aecec804a1674d53b221d77b3a16ead1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2bd8d8380544a878e2300dd65a1bfec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:34:55.913631 4986 tablet_service.cc:1467] Processing CreateTablet for tablet aecec804a1674d53b221d77b3a16ead1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2bd8d8380544a878e2300dd65a1bfec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:34:55.914181 4706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecec804a1674d53b221d77b3a16ead1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:55.914245 4986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecec804a1674d53b221d77b3a16ead1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:55.914245 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecec804a1674d53b221d77b3a16ead1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:55.917243 5077 tablet_bootstrap.cc:492] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: Bootstrap starting.
I20260924 10:34:55.917402 5076 tablet_bootstrap.cc:492] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: Bootstrap starting.
I20260924 10:34:55.917591 5075 tablet_bootstrap.cc:492] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: Bootstrap starting.
I20260924 10:34:55.918332 5077 tablet_bootstrap.cc:654] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:55.918402 5076 tablet_bootstrap.cc:654] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:55.918591 5075 tablet_bootstrap.cc:654] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:55.919823 5076 tablet_bootstrap.cc:492] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: No bootstrap required, opened a new log
I20260924 10:34:55.919900 5076 ts_tablet_manager.cc:1397] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:34:55.920208 5077 tablet_bootstrap.cc:492] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: No bootstrap required, opened a new log
I20260924 10:34:55.920284 5077 ts_tablet_manager.cc:1397] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:34:55.920324 5076 raft_consensus.cc:348] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.920423 5076 raft_consensus.cc:374] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:55.920449 5076 raft_consensus.cc:729] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2632bf24e348278c70983adbc6b703, State: Initialized, Role: FOLLOWER
I20260924 10:34:55.920574 5076 consensus_queue.cc:260] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.920789 5076 ts_tablet_manager.cc:1428] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:55.920746 5077 raft_consensus.cc:348] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.920873 5077 raft_consensus.cc:374] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:55.920914 5077 raft_consensus.cc:729] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9355d899d4e4970ad331e6e09ee4411, State: Initialized, Role: FOLLOWER
I20260924 10:34:55.921033 5082 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:55.921077 5077 consensus_queue.cc:260] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.921270 5075 tablet_bootstrap.cc:492] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: No bootstrap required, opened a new log
I20260924 10:34:55.921345 5075 ts_tablet_manager.cc:1397] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:34:55.921326 5083 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:55.921464 5083 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:55.921787 5075 raft_consensus.cc:348] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.921897 5075 raft_consensus.cc:374] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:55.921928 5075 raft_consensus.cc:729] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92752c45b5ab441d9bf7db71c4bc425e, State: Initialized, Role: FOLLOWER
I20260924 10:34:55.922057 5075 consensus_queue.cc:260] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:55.922318 5075 ts_tablet_manager.cc:1428] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:55.922538 5081 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:55.922678 5081 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:55.922583 5077 ts_tablet_manager.cc:1428] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:34:55.921159 5082 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 10:34:55.934566 5053 tablet.cc:2369] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:34:55.945415 4913 tablet.cc:2369] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:34:55.971850 4773 tablet.cc:2369] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:34:56.269297 5083 raft_consensus.cc:482] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:56.269446 5083 raft_consensus.cc:504] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.269922 5083 leader_election.cc:288] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725)
I20260924 10:34:56.270471 5006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecec804a1674d53b221d77b3a16ead1" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92752c45b5ab441d9bf7db71c4bc425e" is_pre_election: true
I20260924 10:34:56.270471 4726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecec804a1674d53b221d77b3a16ead1" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703" is_pre_election: true
I20260924 10:34:56.270629 4726 raft_consensus.cc:2454] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 0.
I20260924 10:34:56.270629 5006 raft_consensus.cc:2454] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 0.
I20260924 10:34:56.270985 4800 leader_election.cc:302] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f2632bf24e348278c70983adbc6b703, 92752c45b5ab441d9bf7db71c4bc425e, b9355d899d4e4970ad331e6e09ee4411; no voters:
I20260924 10:34:56.271200 5083 raft_consensus.cc:2781] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:56.271276 5083 raft_consensus.cc:482] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:56.271329 5083 raft_consensus.cc:3037] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:56.272324 5083 raft_consensus.cc:504] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.272686 5083 leader_election.cc:288] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 election: Requested vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725)
I20260924 10:34:56.273082 5006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecec804a1674d53b221d77b3a16ead1" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92752c45b5ab441d9bf7db71c4bc425e"
I20260924 10:34:56.273092 4726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecec804a1674d53b221d77b3a16ead1" candidate_uuid: "b9355d899d4e4970ad331e6e09ee4411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703"
I20260924 10:34:56.273247 5006 raft_consensus.cc:3037] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:56.273262 4726 raft_consensus.cc:3037] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:56.274217 5006 raft_consensus.cc:2454] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 1.
I20260924 10:34:56.274227 4726 raft_consensus.cc:2454] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9355d899d4e4970ad331e6e09ee4411 in term 1.
I20260924 10:34:56.274534 4800 leader_election.cc:302] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92752c45b5ab441d9bf7db71c4bc425e, b9355d899d4e4970ad331e6e09ee4411; no voters:
I20260924 10:34:56.274744 5083 raft_consensus.cc:2781] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:56.274837 5083 raft_consensus.cc:686] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [term 1 LEADER]: Becoming Leader. State: Replica: b9355d899d4e4970ad331e6e09ee4411, State: Running, Role: LEADER
I20260924 10:34:56.275010 5083 consensus_queue.cc:237] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.275771 5085 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: TxnStatusTablet state changed. Reason: New leader b9355d899d4e4970ad331e6e09ee4411. Latest consensus state: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:56.275877 5085 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:34:56.276188 5097 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:34:56.276386 5097 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:34:56.276999 4567 catalog_manager.cc:5495] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130). New cstate: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } health_report { overall_health: UNKNOWN } } }
I20260924 10:34:56.278177 5085 consensus_queue.cc:1035] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:56.284566 5085 consensus_queue.cc:1035] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:56.296986 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.292656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36661 (local address 127.0.0.1:59046)
0924 10:34:56.292849 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:56.292866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:56.292985 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:56.293370 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:56.293372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:56.293379 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:56.293472 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.293478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.294527 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.294529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.295144 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.295152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.295788 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.295790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.295864 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.295912 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.296715 (+ 803us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.296729 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 10:34:56.296779 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 10:34:56.296854 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 10:34:56.297022 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.292755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.2.193.130:36661)
0924 10:34:56.293029 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.293037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.293053 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:34:56.293123 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.293126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.293240 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.293356 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:56.293577 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.294419 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.295256 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.295686 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.295788 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.296059 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.296612 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.296723 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.296793 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.296854 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260924 10:34:56.301676 4726 raft_consensus.cc:1261] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Refusing update from remote peer b9355d899d4e4970ad331e6e09ee4411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:56.301715 5006 raft_consensus.cc:1261] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Refusing update from remote peer b9355d899d4e4970ad331e6e09ee4411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:56.302141 5083 consensus_queue.cc:1035] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:56.302376 5085 consensus_queue.cc:1035] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:56.305439 5081 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: TxnStatusTablet state changed. Reason: New leader b9355d899d4e4970ad331e6e09ee4411. Latest consensus state: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:56.305572 5081 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:56.305648 5082 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: TxnStatusTablet state changed. Reason: New leader b9355d899d4e4970ad331e6e09ee4411. Latest consensus state: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:56.305757 5082 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:56.308421 5085 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:56.308562 5085 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:34:56.308790 5082 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:56.308887 5082 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P 4f2632bf24e348278c70983adbc6b703: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:56.309044 5103 mvcc.cc:205] Tried to move back new op lower bound from 7332848010445885440 to 7332848010343870464. Current Snapshot: MvccSnapshot[applied={T|T < 7332848010445885440}]
I20260924 10:34:56.309420 5083 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:56.309535 5083 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P b9355d899d4e4970ad331e6e09ee4411: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:34:56.310402 5081 tablet_replica.cc:439] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9355d899d4e4970ad331e6e09ee4411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } } }
I20260924 10:34:56.310531 5081 tablet_replica.cc:442] T aecec804a1674d53b221d77b3a16ead1 P 92752c45b5ab441d9bf7db71c4bc425e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:34:56.310842 5101 mvcc.cc:205] Tried to move back new op lower bound from 7332848010445885440 to 7332848010343870464. Current Snapshot: MvccSnapshot[applied={T|T < 7332848010445885440}]
I20260924 10:34:56.311614 5100 mvcc.cc:205] Tried to move back new op lower bound from 7332848010445885440 to 7332848010343870464. Current Snapshot: MvccSnapshot[applied={T|T < 7332848010445885440}]
I20260924 10:34:56.320636 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.317003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.2.193.130:36661)
0924 10:34:56.317131 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.317136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.317345 (+ 209us) server_negotiation.cc:407] Connection header received
0924 10:34:56.317437 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.317440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.317492 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.317572 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:56.317837 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.318638 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.319546 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.319890 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.320004 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.320216 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.320297 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.320366 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.320438 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.320495 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:34:56.332247 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.328918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:58605 (local address 127.2.193.130:36661)
0924 10:34:56.329053 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.329059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.329233 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:34:56.329388 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.329391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.329434 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.329511 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:56.329771 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.330267 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.331152 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.331354 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.331457 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.331784 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.331860 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.331935 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.332026 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.332069 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":43}
I20260924 10:34:56.332252 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.328829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36661 (local address 127.2.193.130:58605)
0924 10:34:56.329133 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:56.329145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:56.329283 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:56.329531 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:56.329533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:56.329539 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:56.329674 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.329678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.330377 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.330381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.331036 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.331044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.331456 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.331459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.331573 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.331622 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.331945 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.331953 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:34:56.332010 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 10:34:56.332079 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260924 10:34:56.372109 4569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:56.372272 4569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:56.374744 4569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Sending DeleteTablet for 3 replicas of tablet 1338bbb1ea5b49098e48370787f9513d
I20260924 10:34:56.375569 4706 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338bbb1ea5b49098e48370787f9513d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:56 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:48064
I20260924 10:34:56.376060 5119 tablet_replica.cc:330] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: stopping tablet replica
I20260924 10:34:56.376078 4986 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338bbb1ea5b49098e48370787f9513d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:56 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:44618
I20260924 10:34:56.376197 4569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:34:56.376410 5119 raft_consensus.cc:2229] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 10:34:56.376595 4569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:56.376631 5119 raft_consensus.cc:2258] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:56.376636 4846 tablet_service.cc:1514] Processing DeleteTablet for tablet 1338bbb1ea5b49098e48370787f9513d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:56 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:59030
I20260924 10:34:56.377032 5121 tablet_replica.cc:330] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: stopping tablet replica
I20260924 10:34:56.377329 5121 raft_consensus.cc:2229] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:34:56.377687 5121 raft_consensus.cc:2258] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:56.378535 5119 ts_tablet_manager.cc:1905] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:56.378809 5121 ts_tablet_manager.cc:1905] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:56.381991 4986 tablet_service.cc:1467] Processing CreateTablet for tablet 96d8c3f8e49547a1be0137b78cb69eed (DEFAULT_TABLE table=test-table [id=51efd65f488b45a6b3b4affa6f2143c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:56.381992 4706 tablet_service.cc:1467] Processing CreateTablet for tablet 96d8c3f8e49547a1be0137b78cb69eed (DEFAULT_TABLE table=test-table [id=51efd65f488b45a6b3b4affa6f2143c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:56.382249 4986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d8c3f8e49547a1be0137b78cb69eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:56.382249 4706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d8c3f8e49547a1be0137b78cb69eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:56.383044 5120 tablet_replica.cc:330] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: stopping tablet replica
I20260924 10:34:56.383410 5120 raft_consensus.cc:2229] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:56.383663 5120 raft_consensus.cc:2258] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:56.384538 5076 tablet_bootstrap.cc:492] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: Bootstrap starting.
I20260924 10:34:56.384711 5121 ts_tablet_manager.cc:1918] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:56.384812 5121 log.cc:1196] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/1338bbb1ea5b49098e48370787f9513d
I20260924 10:34:56.385211 5121 ts_tablet_manager.cc:1939] T 1338bbb1ea5b49098e48370787f9513d P b9355d899d4e4970ad331e6e09ee4411: Deleting consensus metadata
I20260924 10:34:56.385354 5075 tablet_bootstrap.cc:492] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: Bootstrap starting.
I20260924 10:34:56.385643 5076 tablet_bootstrap.cc:654] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:56.386193 5075 tablet_bootstrap.cc:654] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:56.386827 5076 tablet_bootstrap.cc:492] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: No bootstrap required, opened a new log
I20260924 10:34:56.386919 5076 ts_tablet_manager.cc:1397] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:34:56.387332 5075 tablet_bootstrap.cc:492] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: No bootstrap required, opened a new log
I20260924 10:34:56.387414 5075 ts_tablet_manager.cc:1397] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:34:56.387420 5076 raft_consensus.cc:348] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.387542 5076 raft_consensus.cc:374] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:56.387601 5076 raft_consensus.cc:729] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2632bf24e348278c70983adbc6b703, State: Initialized, Role: FOLLOWER
I20260924 10:34:56.387717 5076 consensus_queue.cc:260] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.387907 5075 raft_consensus.cc:348] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.388008 5075 raft_consensus.cc:374] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:56.388015 5076 ts_tablet_manager.cc:1428] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:34:56.388046 5075 raft_consensus.cc:729] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92752c45b5ab441d9bf7db71c4bc425e, State: Initialized, Role: FOLLOWER
I20260924 10:34:56.388157 5075 consensus_queue.cc:260] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.388409 5075 ts_tablet_manager.cc:1428] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:56.388808 4556 catalog_manager.cc:4841] TS b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661): tablet 1338bbb1ea5b49098e48370787f9513d (table test-table [id=6ec4dd0cdcb545c0a56c6cf20c9b93d4]) successfully deleted
I20260924 10:34:56.389132 4846 tablet_service.cc:1467] Processing CreateTablet for tablet 96d8c3f8e49547a1be0137b78cb69eed (DEFAULT_TABLE table=test-table [id=51efd65f488b45a6b3b4affa6f2143c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:56.389453 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d8c3f8e49547a1be0137b78cb69eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:56.389674 5119 ts_tablet_manager.cc:1918] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:56.389807 5119 log.cc:1196] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/1338bbb1ea5b49098e48370787f9513d
I20260924 10:34:56.390350 5119 ts_tablet_manager.cc:1939] T 1338bbb1ea5b49098e48370787f9513d P 4f2632bf24e348278c70983adbc6b703: Deleting consensus metadata
I20260924 10:34:56.391639 5077 tablet_bootstrap.cc:492] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: Bootstrap starting.
I20260924 10:34:56.391709 4554 catalog_manager.cc:4841] TS 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679): tablet 1338bbb1ea5b49098e48370787f9513d (table test-table [id=6ec4dd0cdcb545c0a56c6cf20c9b93d4]) successfully deleted
I20260924 10:34:56.392598 5077 tablet_bootstrap.cc:654] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:56.393054 5120 ts_tablet_manager.cc:1905] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:56.393918 5077 tablet_bootstrap.cc:492] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: No bootstrap required, opened a new log
I20260924 10:34:56.394018 5077 ts_tablet_manager.cc:1397] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:56.394400 5077 raft_consensus.cc:348] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.394526 5077 raft_consensus.cc:374] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:56.394567 5077 raft_consensus.cc:729] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9355d899d4e4970ad331e6e09ee4411, State: Initialized, Role: FOLLOWER
I20260924 10:34:56.394711 5077 consensus_queue.cc:260] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.394985 5077 ts_tablet_manager.cc:1428] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:56.396342 5120 ts_tablet_manager.cc:1918] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:56.396435 5120 log.cc:1196] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/1338bbb1ea5b49098e48370787f9513d
I20260924 10:34:56.396724 5120 ts_tablet_manager.cc:1939] T 1338bbb1ea5b49098e48370787f9513d P 92752c45b5ab441d9bf7db71c4bc425e: Deleting consensus metadata
I20260924 10:34:56.398118 4556 catalog_manager.cc:4841] TS 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725): tablet 1338bbb1ea5b49098e48370787f9513d (table test-table [id=6ec4dd0cdcb545c0a56c6cf20c9b93d4]) successfully deleted
I20260924 10:34:56.756836 5082 raft_consensus.cc:482] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:56.756990 5082 raft_consensus.cc:504] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.758102 5082 leader_election.cc:288] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661), 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725)
I20260924 10:34:56.763516 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.758982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:39799 (local address 127.2.193.131:41725)
0924 10:34:56.759242 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.759257 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.759270 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:34:56.759321 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.759324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.759371 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.759436 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:56.759702 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.760422 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.762222 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.762827 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.762954 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.763007 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.763286 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.763356 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260924 10:34:56.763492 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.758469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:46015 (local address 127.2.193.130:36661)
0924 10:34:56.758616 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.758621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.758831 (+ 210us) server_negotiation.cc:407] Connection header received
0924 10:34:56.759014 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.759018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.759088 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.759198 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:56.759543 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.760336 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.762222 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.762827 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.762954 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.763009 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.763286 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.763357 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:34:56.763546 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.758847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41725 (local address 127.2.193.129:39799)
0924 10:34:56.759150 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:56.759163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:56.759249 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:56.759483 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:56.759489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:56.759508 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:56.759605 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.759609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.760539 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.760541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.762086 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.762098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.762956 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.762961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.763126 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.763177 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 10:34:56.763258 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:34:56.763366 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":237,"thread_start_us":88,"threads_started":1}
I20260924 10:34:56.763558 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.758406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36661 (local address 127.2.193.129:46015)
0924 10:34:56.758714 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:56.758728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:56.758886 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:56.759236 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:56.759239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:56.759250 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:56.759394 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.759399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.760451 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.760453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.762086 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.762098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.762981 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.762984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.763105 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.763181 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 10:34:56.763258 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:34:56.763365 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":190,"thread_start_us":114,"threads_started":1}
I20260924 10:34:56.764110 5006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d8c3f8e49547a1be0137b78cb69eed" candidate_uuid: "4f2632bf24e348278c70983adbc6b703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92752c45b5ab441d9bf7db71c4bc425e" is_pre_election: true
I20260924 10:34:56.764271 5006 raft_consensus.cc:2454] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2632bf24e348278c70983adbc6b703 in term 0.
I20260924 10:34:56.764221 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d8c3f8e49547a1be0137b78cb69eed" candidate_uuid: "4f2632bf24e348278c70983adbc6b703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9355d899d4e4970ad331e6e09ee4411" is_pre_election: true
I20260924 10:34:56.764488 4866 raft_consensus.cc:2454] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2632bf24e348278c70983adbc6b703 in term 0.
I20260924 10:34:56.764678 4660 leader_election.cc:302] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2632bf24e348278c70983adbc6b703, 92752c45b5ab441d9bf7db71c4bc425e; no voters:
I20260924 10:34:56.764967 5082 raft_consensus.cc:2781] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:56.765031 5082 raft_consensus.cc:482] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:56.765069 5082 raft_consensus.cc:3037] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:56.766047 5082 raft_consensus.cc:504] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.766388 5082 leader_election.cc:288] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [CANDIDATE]: Term 1 election: Requested vote from peers b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661), 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725)
I20260924 10:34:56.766734 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d8c3f8e49547a1be0137b78cb69eed" candidate_uuid: "4f2632bf24e348278c70983adbc6b703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9355d899d4e4970ad331e6e09ee4411"
I20260924 10:34:56.766801 5006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d8c3f8e49547a1be0137b78cb69eed" candidate_uuid: "4f2632bf24e348278c70983adbc6b703" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92752c45b5ab441d9bf7db71c4bc425e"
I20260924 10:34:56.766856 4866 raft_consensus.cc:3037] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:56.766934 5006 raft_consensus.cc:3037] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:56.767831 4866 raft_consensus.cc:2454] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2632bf24e348278c70983adbc6b703 in term 1.
I20260924 10:34:56.767836 5006 raft_consensus.cc:2454] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2632bf24e348278c70983adbc6b703 in term 1.
I20260924 10:34:56.768133 4660 leader_election.cc:302] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2632bf24e348278c70983adbc6b703, 92752c45b5ab441d9bf7db71c4bc425e; no voters:
I20260924 10:34:56.768291 5082 raft_consensus.cc:2781] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:56.768544 5082 raft_consensus.cc:686] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 1 LEADER]: Becoming Leader. State: Replica: 4f2632bf24e348278c70983adbc6b703, State: Running, Role: LEADER
I20260924 10:34:56.768798 5082 consensus_queue.cc:237] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:56.771169 4569 catalog_manager.cc:5495] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f2632bf24e348278c70983adbc6b703 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "4f2632bf24e348278c70983adbc6b703" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } health_report { overall_health: UNKNOWN } } }
I20260924 10:34:56.848505 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.845057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.2.193.129:34679)
0924 10:34:56.845316 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.845322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.845370 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:34:56.845535 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.845537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.845591 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.845694 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:56.845919 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.846444 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.847209 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.847447 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.847529 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.847737 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.848183 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.848271 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.848323 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.848372 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
I20260924 10:34:56.858850 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.855895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:55111 (local address 127.2.193.130:36661)
0924 10:34:56.856046 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.856052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.856204 (+ 152us) server_negotiation.cc:407] Connection header received
0924 10:34:56.856311 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.856314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.856357 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.856421 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:56.856640 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.857127 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.857988 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.858186 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.858259 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.858504 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.858570 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.858620 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.858697 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.858739 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:34:56.858872 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.855804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36661 (local address 127.2.193.129:55111)
0924 10:34:56.856125 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:56.856138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:56.856227 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:56.856444 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:56.856446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:56.856452 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:56.856557 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.856562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.857250 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.857252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.857888 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.857895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.858280 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.858282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.858348 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.858390 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.858639 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.858644 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:34:56.858690 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 10:34:56.858751 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
I20260924 10:34:56.865604 4866 raft_consensus.cc:1261] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Refusing update from remote peer 4f2632bf24e348278c70983adbc6b703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:56.866042 5006 raft_consensus.cc:1261] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Refusing update from remote peer 4f2632bf24e348278c70983adbc6b703: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:56.866165 5082 consensus_queue.cc:1035] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:56.866449 5128 consensus_queue.cc:1035] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:56.883638 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.880406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:33037 (local address 127.2.193.129:34679)
0924 10:34:56.880540 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:56.880543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:56.880555 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:34:56.880643 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:56.880646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:56.880685 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:56.880781 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:56.881068 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.881698 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.882577 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.882874 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.882982 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.883205 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.883287 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.883361 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:56.883451 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 10:34:56.883504 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:34:56.883658 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:56.880200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:34679 (local address 127.2.193.130:33037)
0924 10:34:56.880443 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:56.880467 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:56.880554 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:56.880822 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:56.880828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:56.880837 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:56.880955 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.880961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.881806 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.881812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.882454 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:56.882469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:56.882969 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:56.882973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:56.883057 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:56.883091 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:56.883375 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:56.883383 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:34:56.883442 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:34:56.883530 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
W20260924 10:34:56.884488 5103 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:34:58.441058 4567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:58.441229 4567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:58.443686 4567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Sending DeleteTablet for 3 replicas of tablet 96d8c3f8e49547a1be0137b78cb69eed
I20260924 10:34:58.444270 4706 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d8c3f8e49547a1be0137b78cb69eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:58 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:48064
I20260924 10:34:58.444378 4846 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d8c3f8e49547a1be0137b78cb69eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:58 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:59030
I20260924 10:34:58.444643 5164 tablet_replica.cc:330] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: stopping tablet replica
I20260924 10:34:58.444790 5164 raft_consensus.cc:2229] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:34:58.445147 5164 raft_consensus.cc:2258] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:58.445812 5164 ts_tablet_manager.cc:1905] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:58.448005 4986 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d8c3f8e49547a1be0137b78cb69eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:58 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:44618
I20260924 10:34:58.448089 5165 tablet_replica.cc:330] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: stopping tablet replica
I20260924 10:34:58.448220 5165 raft_consensus.cc:2229] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:58.448372 5165 raft_consensus.cc:2258] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:58.448844 5164 ts_tablet_manager.cc:1918] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:34:58.448987 5165 ts_tablet_manager.cc:1905] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:58.451015 5164 log.cc:1196] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/96d8c3f8e49547a1be0137b78cb69eed
I20260924 10:34:58.451344 5164 ts_tablet_manager.cc:1939] T 96d8c3f8e49547a1be0137b78cb69eed P 4f2632bf24e348278c70983adbc6b703: Deleting consensus metadata
I20260924 10:34:58.451452 5165 ts_tablet_manager.cc:1918] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:34:58.451525 5165 log.cc:1196] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/96d8c3f8e49547a1be0137b78cb69eed
I20260924 10:34:58.451810 5165 ts_tablet_manager.cc:1939] T 96d8c3f8e49547a1be0137b78cb69eed P b9355d899d4e4970ad331e6e09ee4411: Deleting consensus metadata
I20260924 10:34:58.452159 4554 catalog_manager.cc:4841] TS 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679): tablet 96d8c3f8e49547a1be0137b78cb69eed (table test-table [id=51efd65f488b45a6b3b4affa6f2143c5]) successfully deleted
I20260924 10:34:58.452492 4556 catalog_manager.cc:4841] TS b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661): tablet 96d8c3f8e49547a1be0137b78cb69eed (table test-table [id=51efd65f488b45a6b3b4affa6f2143c5]) successfully deleted
I20260924 10:34:58.453630 5167 tablet_replica.cc:330] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: stopping tablet replica
I20260924 10:34:58.453796 5167 raft_consensus.cc:2229] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:58.453951 5167 raft_consensus.cc:2258] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:58.454679 5167 ts_tablet_manager.cc:1905] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:58.457021 5167 ts_tablet_manager.cc:1918] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:34:58.457093 5167 log.cc:1196] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/96d8c3f8e49547a1be0137b78cb69eed
I20260924 10:34:58.457396 5167 ts_tablet_manager.cc:1939] T 96d8c3f8e49547a1be0137b78cb69eed P 92752c45b5ab441d9bf7db71c4bc425e: Deleting consensus metadata
I20260924 10:34:58.458411 4556 catalog_manager.cc:4841] TS 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725): tablet 96d8c3f8e49547a1be0137b78cb69eed (table test-table [id=51efd65f488b45a6b3b4affa6f2143c5]) successfully deleted
Sep 24 10:34:58 dist-test-slave-8wc7 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:34:58 dist-test-slave-8wc7 krb5kdc[4524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:34:58.484176 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.468466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.2.193.190:36467)
0924 10:34:58.468780 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:58.468786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:58.468802 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:34:58.468949 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:58.468951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:58.468998 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:58.469082 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:34:58.469877 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.470468 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.471467 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.471722 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.471821 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.481154 (+ 9333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:34:58.481158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:34:58.481160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:34:58.481258 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:34:58.482649 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:58.483061 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:58.483063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:58.483065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:58.483140 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:34:58.483362 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:34:58.483364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:34:58.483365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:34:58.483546 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:34:58.483645 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:58.483855 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 10:34:58.483963 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":138,"threads_started":1}
I20260924 10:34:58.486769 4567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:34:58.487226 4567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:58.492250 4846 tablet_service.cc:1467] Processing CreateTablet for tablet ad3f42bc3ed64c6d92a3fddcb289bcac (DEFAULT_TABLE table=test-table [id=29980c39e1124efa84acbf3c07f0a0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:58.492215 4706 tablet_service.cc:1467] Processing CreateTablet for tablet ad3f42bc3ed64c6d92a3fddcb289bcac (DEFAULT_TABLE table=test-table [id=29980c39e1124efa84acbf3c07f0a0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:58.492589 4706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3f42bc3ed64c6d92a3fddcb289bcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:58.492590 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3f42bc3ed64c6d92a3fddcb289bcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:58.495666 4986 tablet_service.cc:1467] Processing CreateTablet for tablet ad3f42bc3ed64c6d92a3fddcb289bcac (DEFAULT_TABLE table=test-table [id=29980c39e1124efa84acbf3c07f0a0d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:58.496044 4986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3f42bc3ed64c6d92a3fddcb289bcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:58.497435 5175 tablet_bootstrap.cc:492] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: Bootstrap starting.
I20260924 10:34:58.498377 5175 tablet_bootstrap.cc:654] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:58.498668 5177 tablet_bootstrap.cc:492] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: Bootstrap starting.
I20260924 10:34:58.499050 5176 tablet_bootstrap.cc:492] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: Bootstrap starting.
I20260924 10:34:58.499630 5175 tablet_bootstrap.cc:492] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: No bootstrap required, opened a new log
I20260924 10:34:58.499765 5175 ts_tablet_manager.cc:1397] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:58.499922 5176 tablet_bootstrap.cc:654] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:58.500046 5177 tablet_bootstrap.cc:654] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:58.500279 5175 raft_consensus.cc:348] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.500422 5175 raft_consensus.cc:374] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:58.500486 5175 raft_consensus.cc:729] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9355d899d4e4970ad331e6e09ee4411, State: Initialized, Role: FOLLOWER
I20260924 10:34:58.500624 5175 consensus_queue.cc:260] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.501130 5175 ts_tablet_manager.cc:1428] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:58.501240 5176 tablet_bootstrap.cc:492] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: No bootstrap required, opened a new log
I20260924 10:34:58.501317 5176 ts_tablet_manager.cc:1397] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:34:58.501758 5176 raft_consensus.cc:348] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.501874 5176 raft_consensus.cc:374] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:58.501917 5176 raft_consensus.cc:729] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2632bf24e348278c70983adbc6b703, State: Initialized, Role: FOLLOWER
I20260924 10:34:58.502063 5176 consensus_queue.cc:260] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.502332 5176 ts_tablet_manager.cc:1428] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:34:58.503104 5177 tablet_bootstrap.cc:492] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: No bootstrap required, opened a new log
I20260924 10:34:58.503208 5177 ts_tablet_manager.cc:1397] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 10:34:58.503768 5177 raft_consensus.cc:348] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.503898 5177 raft_consensus.cc:374] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:58.503937 5177 raft_consensus.cc:729] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92752c45b5ab441d9bf7db71c4bc425e, State: Initialized, Role: FOLLOWER
I20260924 10:34:58.504117 5177 consensus_queue.cc:260] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.504562 5177 ts_tablet_manager.cc:1428] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 10:34:58.735076 5181 raft_consensus.cc:482] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:58.735231 5181 raft_consensus.cc:504] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.736403 5181 leader_election.cc:288] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661)
I20260924 10:34:58.741991 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.736772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:33027 (local address 127.2.193.130:36661)
0924 10:34:58.737114 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:58.737124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:58.737346 (+ 222us) server_negotiation.cc:407] Connection header received
0924 10:34:58.737450 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:58.737453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:58.737499 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:58.737580 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:58.737811 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.738640 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.740565 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.741243 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.741372 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.741426 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:58.741755 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 10:34:58.741828 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":101,"threads_started":1}
I20260924 10:34:58.741986 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.736735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:50477 (local address 127.2.193.129:34679)
0924 10:34:58.737055 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:58.737065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:58.737083 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:34:58.737290 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:58.737294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:58.737373 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:58.737459 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:34:58.737811 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.738640 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.740565 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.741243 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.741372 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.741426 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:58.741755 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 10:34:58.741828 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260924 10:34:58.742044 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.736683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36661 (local address 127.2.193.131:33027)
0924 10:34:58.737255 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:58.737266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:58.737356 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:58.737598 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:58.737601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:58.737613 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:58.737701 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.737710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.738777 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.738780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.740422 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.740437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.741373 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.741377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.741565 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.741643 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 10:34:58.741726 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:34:58.741843 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":297,"thread_start_us":126,"threads_started":1}
I20260924 10:34:58.742053 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.736600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:34679 (local address 127.2.193.131:50477)
0924 10:34:58.737001 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:58.737018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:58.737134 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:58.737498 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:58.737501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:58.737514 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:34:58.737669 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.737679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.738762 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.738767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.740422 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.740437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.741373 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.741377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.741571 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.741642 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:34:58.741726 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:34:58.741843 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":210,"thread_start_us":126,"threads_started":1}
I20260924 10:34:58.742697 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3f42bc3ed64c6d92a3fddcb289bcac" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9355d899d4e4970ad331e6e09ee4411" is_pre_election: true
I20260924 10:34:58.742683 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3f42bc3ed64c6d92a3fddcb289bcac" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703" is_pre_election: true
I20260924 10:34:58.742852 4866 raft_consensus.cc:2454] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 0.
I20260924 10:34:58.742853 4725 raft_consensus.cc:2454] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 0.
I20260924 10:34:58.743307 4938 leader_election.cc:302] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f2632bf24e348278c70983adbc6b703, 92752c45b5ab441d9bf7db71c4bc425e, b9355d899d4e4970ad331e6e09ee4411; no voters:
I20260924 10:34:58.743577 5181 raft_consensus.cc:2781] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:58.743637 5181 raft_consensus.cc:482] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:58.743666 5181 raft_consensus.cc:3037] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:58.744706 5181 raft_consensus.cc:504] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.745081 5181 leader_election.cc:288] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 election: Requested vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661)
I20260924 10:34:58.745496 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3f42bc3ed64c6d92a3fddcb289bcac" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9355d899d4e4970ad331e6e09ee4411"
I20260924 10:34:58.745515 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3f42bc3ed64c6d92a3fddcb289bcac" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703"
I20260924 10:34:58.745626 4725 raft_consensus.cc:3037] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:58.745626 4866 raft_consensus.cc:3037] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:58.746766 4725 raft_consensus.cc:2454] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 1.
I20260924 10:34:58.746766 4866 raft_consensus.cc:2454] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 1.
I20260924 10:34:58.747094 4940 leader_election.cc:302] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92752c45b5ab441d9bf7db71c4bc425e, b9355d899d4e4970ad331e6e09ee4411; no voters:
I20260924 10:34:58.747308 5181 raft_consensus.cc:2781] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:58.747519 5181 raft_consensus.cc:686] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 1 LEADER]: Becoming Leader. State: Replica: 92752c45b5ab441d9bf7db71c4bc425e, State: Running, Role: LEADER
I20260924 10:34:58.747772 5181 consensus_queue.cc:237] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.750309 4567 catalog_manager.cc:5495] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131). New cstate: current_term: 1 leader_uuid: "92752c45b5ab441d9bf7db71c4bc425e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } health_report { overall_health: HEALTHY } } }
I20260924 10:34:58.797884 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.794165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.2.193.131:41725)
0924 10:34:58.794431 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:58.794438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:58.794450 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:34:58.794527 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:58.794529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:58.794590 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:58.794698 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:58.794925 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.795472 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.796403 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.796650 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.796786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.797047 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:58.797548 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:58.797637 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:58.797702 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 10:34:58.797753 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
I20260924 10:34:58.807806 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.804927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:39883 (local address 127.2.193.130:36661)
0924 10:34:58.805081 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:58.805085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:58.805099 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:34:58.805249 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:58.805252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:58.805295 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:58.805357 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:58.805596 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.806088 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.806884 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.807089 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.807183 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.807427 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:58.807497 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:58.807549 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:58.807650 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 10:34:58.807695 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 10:34:58.807826 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.804698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36661 (local address 127.2.193.131:39883)
0924 10:34:58.805013 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:58.805026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:58.805109 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:58.805382 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:58.805385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:58.805392 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:58.805502 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.805506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.806174 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.806176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.806783 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.806791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.807187 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.807189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.807267 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.807311 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:58.807568 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:58.807573 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:34:58.807641 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:34:58.807702 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":117,"threads_started":1}
I20260924 10:34:58.814103 4725 raft_consensus.cc:1261] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Refusing update from remote peer 92752c45b5ab441d9bf7db71c4bc425e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:58.814103 4866 raft_consensus.cc:1261] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Refusing update from remote peer 92752c45b5ab441d9bf7db71c4bc425e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:58.814575 5181 consensus_queue.cc:1035] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:58.814767 5186 consensus_queue.cc:1035] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:58.831847 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.828477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:35585 (local address 127.2.193.131:41725)
0924 10:34:58.828603 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:34:58.828608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:34:58.828799 (+ 191us) server_negotiation.cc:407] Connection header received
0924 10:34:58.828926 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:34:58.828929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:34:58.828980 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:34:58.829055 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:34:58.829461 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.829959 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.830811 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.831056 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.831162 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.831406 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:34:58.831488 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:34:58.831567 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:34:58.831659 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 10:34:58.831709 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:34:58.831864 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:34:58.828420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41725 (local address 127.2.193.130:35585)
0924 10:34:58.828699 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:34:58.828714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:34:58.828822 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:34:58.829084 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:34:58.829087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:34:58.829095 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:34:58.829236 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.829242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.830079 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.830086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.830692 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:34:58.830699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:34:58.831170 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:34:58.831173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:34:58.831257 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:34:58.831292 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:34:58.831570 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:34:58.831579 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:34:58.831641 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:34:58.831721 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":124,"threads_started":1}
I20260924 10:34:58.855943 4569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:58.856133 4569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:34:58.858368 4569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Sending DeleteTablet for 3 replicas of tablet ad3f42bc3ed64c6d92a3fddcb289bcac
I20260924 10:34:58.859016 4986 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3f42bc3ed64c6d92a3fddcb289bcac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:58 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:44618
I20260924 10:34:58.859110 4706 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3f42bc3ed64c6d92a3fddcb289bcac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:58 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:48064
I20260924 10:34:58.859328 5167 tablet_replica.cc:330] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: stopping tablet replica
I20260924 10:34:58.859349 4846 tablet_service.cc:1514] Processing DeleteTablet for tablet ad3f42bc3ed64c6d92a3fddcb289bcac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:34:58 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:59030
I20260924 10:34:58.859455 5167 raft_consensus.cc:2229] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:34:58.859560 5165 tablet_replica.cc:330] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: stopping tablet replica
I20260924 10:34:58.859704 5165 raft_consensus.cc:2229] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:58.859769 5167 raft_consensus.cc:2258] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:58.859853 5165 raft_consensus.cc:2258] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:58.859972 5164 tablet_replica.cc:330] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: stopping tablet replica
I20260924 10:34:58.860091 5164 raft_consensus.cc:2229] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:34:58.860210 5164 raft_consensus.cc:2258] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:34:58.860669 5165 ts_tablet_manager.cc:1905] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:58.860883 5164 ts_tablet_manager.cc:1905] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:58.860972 5167 ts_tablet_manager.cc:1905] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:34:58.863323 5165 ts_tablet_manager.cc:1918] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:58.863323 5164 ts_tablet_manager.cc:1918] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:58.863397 5165 log.cc:1196] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/ad3f42bc3ed64c6d92a3fddcb289bcac
I20260924 10:34:58.863401 5164 log.cc:1196] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/ad3f42bc3ed64c6d92a3fddcb289bcac
I20260924 10:34:58.863550 5167 ts_tablet_manager.cc:1918] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:34:58.863619 5167 log.cc:1196] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/ad3f42bc3ed64c6d92a3fddcb289bcac
I20260924 10:34:58.863670 5164 ts_tablet_manager.cc:1939] T ad3f42bc3ed64c6d92a3fddcb289bcac P 4f2632bf24e348278c70983adbc6b703: Deleting consensus metadata
I20260924 10:34:58.863670 5165 ts_tablet_manager.cc:1939] T ad3f42bc3ed64c6d92a3fddcb289bcac P b9355d899d4e4970ad331e6e09ee4411: Deleting consensus metadata
I20260924 10:34:58.863905 5167 ts_tablet_manager.cc:1939] T ad3f42bc3ed64c6d92a3fddcb289bcac P 92752c45b5ab441d9bf7db71c4bc425e: Deleting consensus metadata
I20260924 10:34:58.864351 4556 catalog_manager.cc:4841] TS b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661): tablet ad3f42bc3ed64c6d92a3fddcb289bcac (table test-table [id=29980c39e1124efa84acbf3c07f0a0d9]) successfully deleted
I20260924 10:34:58.864697 4556 catalog_manager.cc:4841] TS 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725): tablet ad3f42bc3ed64c6d92a3fddcb289bcac (table test-table [id=29980c39e1124efa84acbf3c07f0a0d9]) successfully deleted
I20260924 10:34:58.864351 4554 catalog_manager.cc:4841] TS 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679): tablet ad3f42bc3ed64c6d92a3fddcb289bcac (table test-table [id=29980c39e1124efa84acbf3c07f0a0d9]) successfully deleted
I20260924 10:34:58.865267 4569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:34:58.865703 4569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:34:58.870333 4706 tablet_service.cc:1467] Processing CreateTablet for tablet c198569cf33443a0b9d95b87ea1754b5 (DEFAULT_TABLE table=test-table [id=7bb97adf31ec4ef384463de7f6118002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:58.870456 4846 tablet_service.cc:1467] Processing CreateTablet for tablet c198569cf33443a0b9d95b87ea1754b5 (DEFAULT_TABLE table=test-table [id=7bb97adf31ec4ef384463de7f6118002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:58.870613 4706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c198569cf33443a0b9d95b87ea1754b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:58.870601 4986 tablet_service.cc:1467] Processing CreateTablet for tablet c198569cf33443a0b9d95b87ea1754b5 (DEFAULT_TABLE table=test-table [id=7bb97adf31ec4ef384463de7f6118002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:34:58.870714 4846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c198569cf33443a0b9d95b87ea1754b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:58.870822 4986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c198569cf33443a0b9d95b87ea1754b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:34:58.872923 5176 tablet_bootstrap.cc:492] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703: Bootstrap starting.
I20260924 10:34:58.873153 5175 tablet_bootstrap.cc:492] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: Bootstrap starting.
I20260924 10:34:58.873845 5176 tablet_bootstrap.cc:654] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:58.873987 5175 tablet_bootstrap.cc:654] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:58.874387 5177 tablet_bootstrap.cc:492] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: Bootstrap starting.
I20260924 10:34:58.875345 5175 tablet_bootstrap.cc:492] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: No bootstrap required, opened a new log
I20260924 10:34:58.875361 5177 tablet_bootstrap.cc:654] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: Neither blocks nor log segments found. Creating new log.
I20260924 10:34:58.875463 5175 ts_tablet_manager.cc:1397] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:34:58.875941 5175 raft_consensus.cc:348] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.876071 5175 raft_consensus.cc:374] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:58.876111 5175 raft_consensus.cc:729] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9355d899d4e4970ad331e6e09ee4411, State: Initialized, Role: FOLLOWER
I20260924 10:34:58.876232 5175 consensus_queue.cc:260] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.876346 5176 tablet_bootstrap.cc:492] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703: No bootstrap required, opened a new log
I20260924 10:34:58.876430 5176 ts_tablet_manager.cc:1397] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:34:58.876518 5175 ts_tablet_manager.cc:1428] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:34:58.876791 5176 raft_consensus.cc:348] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.876906 5176 raft_consensus.cc:374] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:58.876937 5176 raft_consensus.cc:729] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2632bf24e348278c70983adbc6b703, State: Initialized, Role: FOLLOWER
I20260924 10:34:58.877044 5176 consensus_queue.cc:260] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.877308 5176 ts_tablet_manager.cc:1428] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:58.878474 5177 tablet_bootstrap.cc:492] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: No bootstrap required, opened a new log
I20260924 10:34:58.878579 5177 ts_tablet_manager.cc:1397] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:34:58.879032 5177 raft_consensus.cc:348] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.879161 5177 raft_consensus.cc:374] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:34:58.879194 5177 raft_consensus.cc:729] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92752c45b5ab441d9bf7db71c4bc425e, State: Initialized, Role: FOLLOWER
I20260924 10:34:58.879309 5177 consensus_queue.cc:260] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.879560 5177 ts_tablet_manager.cc:1428] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:34:58.941321 5181 raft_consensus.cc:482] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:34:58.941498 5181 raft_consensus.cc:504] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.941947 5181 leader_election.cc:288] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661)
I20260924 10:34:58.942385 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c198569cf33443a0b9d95b87ea1754b5" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9355d899d4e4970ad331e6e09ee4411" is_pre_election: true
I20260924 10:34:58.942384 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c198569cf33443a0b9d95b87ea1754b5" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703" is_pre_election: true
I20260924 10:34:58.942554 4725 raft_consensus.cc:2454] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 0.
I20260924 10:34:58.942564 4866 raft_consensus.cc:2454] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 0.
I20260924 10:34:58.942883 4940 leader_election.cc:302] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92752c45b5ab441d9bf7db71c4bc425e, b9355d899d4e4970ad331e6e09ee4411; no voters:
I20260924 10:34:58.943091 5181 raft_consensus.cc:2781] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:34:58.943169 5181 raft_consensus.cc:482] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:34:58.943224 5181 raft_consensus.cc:3037] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:58.944159 5181 raft_consensus.cc:504] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.944546 5181 leader_election.cc:288] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 election: Requested vote from peers 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679), b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661)
I20260924 10:34:58.944916 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c198569cf33443a0b9d95b87ea1754b5" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2632bf24e348278c70983adbc6b703"
I20260924 10:34:58.944916 4866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c198569cf33443a0b9d95b87ea1754b5" candidate_uuid: "92752c45b5ab441d9bf7db71c4bc425e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9355d899d4e4970ad331e6e09ee4411"
I20260924 10:34:58.945055 4725 raft_consensus.cc:3037] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:58.945066 4866 raft_consensus.cc:3037] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:34:58.946056 4725 raft_consensus.cc:2454] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 1.
I20260924 10:34:58.946264 4866 raft_consensus.cc:2454] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92752c45b5ab441d9bf7db71c4bc425e in term 1.
I20260924 10:34:58.946380 4938 leader_election.cc:302] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2632bf24e348278c70983adbc6b703, 92752c45b5ab441d9bf7db71c4bc425e; no voters:
I20260924 10:34:58.946574 5181 raft_consensus.cc:2781] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:34:58.946671 5181 raft_consensus.cc:686] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 1 LEADER]: Becoming Leader. State: Replica: 92752c45b5ab441d9bf7db71c4bc425e, State: Running, Role: LEADER
I20260924 10:34:58.946822 5181 consensus_queue.cc:237] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } }
I20260924 10:34:58.948474 4568 catalog_manager.cc:5495] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131). New cstate: current_term: 1 leader_uuid: "92752c45b5ab441d9bf7db71c4bc425e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92752c45b5ab441d9bf7db71c4bc425e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41725 } health_report { overall_health: HEALTHY } } }
I20260924 10:34:58.967576 4866 raft_consensus.cc:1261] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Refusing update from remote peer 92752c45b5ab441d9bf7db71c4bc425e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:58.967586 4725 raft_consensus.cc:1261] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703 [term 1 FOLLOWER]: Refusing update from remote peer 92752c45b5ab441d9bf7db71c4bc425e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:34:58.968034 5195 consensus_queue.cc:1035] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2632bf24e348278c70983adbc6b703" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:34:58.968415 5181 consensus_queue.cc:1035] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9355d899d4e4970ad331e6e09ee4411" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:34:58.984736 5147 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:35:00.544070 4568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:00.544269 4568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:00.546602 4568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfefde44165a4bc0ae9b317cc97d3f8c: Sending DeleteTablet for 3 replicas of tablet c198569cf33443a0b9d95b87ea1754b5
I20260924 10:35:00.547160 4706 tablet_service.cc:1514] Processing DeleteTablet for tablet c198569cf33443a0b9d95b87ea1754b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:00 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:48064
I20260924 10:35:00.547215 4846 tablet_service.cc:1514] Processing DeleteTablet for tablet c198569cf33443a0b9d95b87ea1754b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:00 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:59030
I20260924 10:35:00.547547 5228 tablet_replica.cc:330] T c198569cf33443a0b9d95b87ea1754b5 P 4f2632bf24e348278c70983adbc6b703: stopping tablet replica
I20260924 10:35:00.547644 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 4641
I20260924 10:35:00.548157 4986 tablet_service.cc:1514] Processing DeleteTablet for tablet c198569cf33443a0b9d95b87ea1754b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:00 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:44618
I20260924 10:35:00.548480 5230 tablet_replica.cc:330] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: stopping tablet replica
I20260924 10:35:00.548672 5230 raft_consensus.cc:2229] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:00.548945 5230 raft_consensus.cc:2258] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:00.549346 5229 tablet_replica.cc:330] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: stopping tablet replica
I20260924 10:35:00.549506 5229 raft_consensus.cc:2229] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:00.549642 5229 raft_consensus.cc:2258] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:00.550254 5229 ts_tablet_manager.cc:1905] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:00.552337 5229 ts_tablet_manager.cc:1918] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:35:00.552433 5229 log.cc:1196] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/c198569cf33443a0b9d95b87ea1754b5
I20260924 10:35:00.552706 5229 ts_tablet_manager.cc:1939] T c198569cf33443a0b9d95b87ea1754b5 P b9355d899d4e4970ad331e6e09ee4411: Deleting consensus metadata
I20260924 10:35:00.553468 4556 catalog_manager.cc:4841] TS b9355d899d4e4970ad331e6e09ee4411 (127.2.193.130:36661): tablet c198569cf33443a0b9d95b87ea1754b5 (table test-table [id=7bb97adf31ec4ef384463de7f6118002]) successfully deleted
I20260924 10:35:00.553776 5230 ts_tablet_manager.cc:1905] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:35:00.556135 4554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:34679 (error 108)
I20260924 10:35:00.556896 5230 ts_tablet_manager.cc:1918] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:35:00.557075 5230 log.cc:1196] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/c198569cf33443a0b9d95b87ea1754b5
I20260924 10:35:00.557111 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 4781
I20260924 10:35:00.557333 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:00.556852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:34679 (local address 127.0.0.1:42630)
0924 10:35:00.557086 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:00.557213 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260924 10:35:00.557909 5230 ts_tablet_manager.cc:1939] T c198569cf33443a0b9d95b87ea1754b5 P 92752c45b5ab441d9bf7db71c4bc425e: Deleting consensus metadata
W20260924 10:35:00.557984 4554 catalog_manager.cc:4584] TS 4f2632bf24e348278c70983adbc6b703 (127.2.193.129:34679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c198569cf33443a0b9d95b87ea1754b5: Network error: Client connection negotiation failed: client connection to 127.2.193.129:34679: connect: Connection refused (error 111)
I20260924 10:35:00.558698 4556 catalog_manager.cc:4841] TS 92752c45b5ab441d9bf7db71c4bc425e (127.2.193.131:41725): tablet c198569cf33443a0b9d95b87ea1754b5 (table test-table [id=7bb97adf31ec4ef384463de7f6118002]) successfully deleted
I20260924 10:35:00.565393 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 4921
I20260924 10:35:00.571986 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 4537
2026-09-24T10:35:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6979 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:00 dist-test-slave-8wc7 krb5kdc[5235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:00 dist-test-slave-8wc7 krb5kdc[5235](info): set up 2 sockets
Sep 24 10:35:00 dist-test-slave-8wc7 krb5kdc[5235](info): commencing operation
krb5kdc: starting...
W20260924 10:35:01.605129 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:01 dist-test-slave-8wc7 krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:01.729590 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:35297
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:01.843494 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:01.843786 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:01.843827 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:01.848093 5248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:01.848163 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:01.848182 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:01.848201 5248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:01.848245 5248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:01.853782 5248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:35297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:01.854918 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:01.856035 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:01.865082 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:01.865103 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:01.865129 5248 server_base.cc:1034] running on GCE node
W20260924 10:35:01.865140 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:01.866037 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:01.867149 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:01.868343 5248 hybrid_clock.cc:648] HybridClock initialized: now 1790246101868329 us; error 31 us; skew 500 ppm
Sep 24 10:35:01 dist-test-slave-8wc7 krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246101, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:01.871127 5248 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:01.872020 5248 webserver.cc:460] Webserver started at http://127.2.193.190:35981/ using document root <none> and password file <none>
I20260924 10:35:01.872555 5248 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:01.872618 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:01.872802 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:01.874939 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "638a9548b28c485ba4e280468296d867"
format_stamp: "Formatted at 2026-09-24 10:35:01 on dist-test-slave-8wc7"
server_key: "456b4abdc658af8ce582457743dc344a"
server_key_iv: "f65e88714cb5d6f85c1fa20240d5f6e7"
server_key_version: "encryptionkey@0"
I20260924 10:35:01.875514 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "638a9548b28c485ba4e280468296d867"
format_stamp: "Formatted at 2026-09-24 10:35:01 on dist-test-slave-8wc7"
server_key: "456b4abdc658af8ce582457743dc344a"
server_key_iv: "f65e88714cb5d6f85c1fa20240d5f6e7"
server_key_version: "encryptionkey@0"
I20260924 10:35:01.879376 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 10:35:01.881865 5263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:01.883148 5248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:35:01.883270 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "638a9548b28c485ba4e280468296d867"
format_stamp: "Formatted at 2026-09-24 10:35:01 on dist-test-slave-8wc7"
server_key: "456b4abdc658af8ce582457743dc344a"
server_key_iv: "f65e88714cb5d6f85c1fa20240d5f6e7"
server_key_version: "encryptionkey@0"
I20260924 10:35:01.883368 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:01.894691 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:01.897701 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:01.897868 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:01.905090 5248 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:35297
I20260924 10:35:01.905107 5315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:35297 every 8 connection(s)
I20260924 10:35:01.906199 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:01.908960 5316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:01.914675 5316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867: Bootstrap starting.
I20260924 10:35:01.916287 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 5248
I20260924 10:35:01.916383 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:01.916669 2822 external_mini_cluster.cc:1428] Setting key 6f416097ec7285a6cfa86f5d69f61e60
I20260924 10:35:01.917503 5316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:01.918412 5316 log.cc:824] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:01.920686 5316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867: No bootstrap required, opened a new log
I20260924 10:35:01.924413 5316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638a9548b28c485ba4e280468296d867" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35297 } }
I20260924 10:35:01.924688 5316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:01.924785 5316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638a9548b28c485ba4e280468296d867, State: Initialized, Role: FOLLOWER
I20260924 10:35:01.925346 5316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638a9548b28c485ba4e280468296d867" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35297 } }
I20260924 10:35:01.925509 5316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:01.925583 5316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 24 10:35:01 dist-test-slave-8wc7 krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:01.925686 5316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:01.926896 5316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638a9548b28c485ba4e280468296d867" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35297 } }
I20260924 10:35:01.927291 5316 leader_election.cc:302] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638a9548b28c485ba4e280468296d867; no voters:
I20260924 10:35:01.927713 5316 leader_election.cc:288] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:01.927834 5321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:01.928069 5321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [term 1 LEADER]: Becoming Leader. State: Replica: 638a9548b28c485ba4e280468296d867, State: Running, Role: LEADER
I20260924 10:35:01.928458 5321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638a9548b28c485ba4e280468296d867" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35297 } }
I20260924 10:35:01.929134 5316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:01.929744 5321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638a9548b28c485ba4e280468296d867. Latest consensus state: current_term: 1 leader_uuid: "638a9548b28c485ba4e280468296d867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638a9548b28c485ba4e280468296d867" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35297 } } }
I20260924 10:35:01.929968 5321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:01.930398 5329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:01.930781 5323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638a9548b28c485ba4e280468296d867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638a9548b28c485ba4e280468296d867" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35297 } } }
I20260924 10:35:01.930930 5323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:01.933386 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:01.917880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.2.193.190:35297)
0924 10:35:01.918393 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:01.918413 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:01.918460 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:35:01.919331 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:01.919356 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:01.919782 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:01.920165 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:01.920848 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:01.921942 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:01.922825 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:01.923314 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:01.923509 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:01.926431 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:01.926436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:01.926452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:01.926481 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:01.928602 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:01.929118 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:01.929124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:01.929130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:01.929299 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:01.929653 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:01.929656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:01.929658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:01.930237 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:01.931884 (+ 1647us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:01.932048 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 10:35:01.932365 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":153,"threads_started":1}
I20260924 10:35:01.934314 5329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:01.940544 5329 catalog_manager.cc:1347] Generated new cluster ID: c95a3fdaf0f3471d9e1bcdf3060250e4
I20260924 10:35:01.940678 5329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:01.978576 5329 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:01.979918 5329 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:01.985639 5329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 638a9548b28c485ba4e280468296d867: Generated new TSK 0
I20260924 10:35:01.986505 5329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:35:02.016136 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:35297
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:02.132131 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:02.132422 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:02.132469 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:02.136653 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:02.136736 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:02.136796 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:02.142491 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:35297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:02.144032 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:02.145480 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:02.155948 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:02.155952 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:02.155952 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:02.156198 5342 server_base.cc:1034] running on GCE node
I20260924 10:35:02.156780 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:02.157536 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:02.158738 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790246102158724 us; error 40 us; skew 500 ppm
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246102, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:02.161645 5342 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:35:02.162644 5342 webserver.cc:460] Webserver started at http://127.2.193.129:45655/ using document root <none> and password file <none>
I20260924 10:35:02.163239 5342 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:02.163336 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:02.163595 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:02.165843 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "b990bb74c68c42739cd46d6b76897678"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "8f2cc65748b1b16c551639da8ef028a5"
server_key_iv: "089a7f0816823c734f25c9221c8a8885"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.166451 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "b990bb74c68c42739cd46d6b76897678"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "8f2cc65748b1b16c551639da8ef028a5"
server_key_iv: "089a7f0816823c734f25c9221c8a8885"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.170410 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 10:35:02.173079 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.174500 5342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:02.174659 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "b990bb74c68c42739cd46d6b76897678"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "8f2cc65748b1b16c551639da8ef028a5"
server_key_iv: "089a7f0816823c734f25c9221c8a8885"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.174790 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:02.224249 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:02.227496 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:02.227721 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:02.228350 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:02.229241 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:02.229315 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.229396 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:02.229449 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.238857 5342 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:40767
I20260924 10:35:02.238875 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:40767 every 8 connection(s)
I20260924 10:35:02.239811 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:02.243676 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 5342
I20260924 10:35:02.243794 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:02.244143 2822 external_mini_cluster.cc:1428] Setting key a506ec7d629b9b467f3c13f0a4da028f
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246102, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:35:02.257359 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:02.241749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:42095 (local address 127.2.193.190:35297)
0924 10:35:02.241935 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:02.241941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:02.243182 (+ 1241us) server_negotiation.cc:407] Connection header received
0924 10:35:02.245197 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:02.245201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:02.245265 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:02.245363 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:02.247277 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.248170 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.250130 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.250368 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.250465 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:02.253565 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:02.253572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:02.253577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:02.253614 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:02.255305 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:02.255799 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:02.255802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:02.255803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:02.255878 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:02.256174 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:02.256176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:02.256177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:02.256416 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:02.256504 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:02.257005 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 10:35:02.257117 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:02.258510 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:02.242336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:35297 (local address 127.2.193.129:42095)
0924 10:35:02.242976 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:02.243057 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:02.244880 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:02.245654 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:02.245665 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:02.246262 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:02.247068 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:02.247089 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.248359 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.248364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:02.249976 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:02.249992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.250560 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.250567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:02.250826 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:02.251506 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:02.251533 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:02.253365 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:02.255427 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:02.255433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:02.255446 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:02.255705 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:02.255962 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:02.255963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:02.255965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:02.256098 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:02.256522 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:02.256535 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:02.256788 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 10:35:02.256962 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 10:35:02.257413 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":154,"threads_started":1}
I20260924 10:35:02.259810 5471 heartbeater.cc:347] Connected to a master server at 127.2.193.190:35297
I20260924 10:35:02.260082 5471 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:02.260594 5471 heartbeater.cc:511] Master 127.2.193.190:35297 requested a full tablet report, sending...
I20260924 10:35:02.262245 5280 ts_manager.cc:194] Registered new tserver with Master: b990bb74c68c42739cd46d6b76897678 (127.2.193.129:40767)
I20260924 10:35:02.263401 5280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:42095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:35:02.275825 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:35297
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:02.391489 5476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:02.391785 5476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:02.391867 5476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:02.396250 5476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:02.396330 5476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:02.396420 5476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:02.402215 5476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:35297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:02.403358 5476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:02.404424 5476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:02.414340 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:02.414409 5476 server_base.cc:1034] running on GCE node
W20260924 10:35:02.414350 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:02.414355 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:02.415145 5476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:02.415773 5476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:02.416939 5476 hybrid_clock.cc:648] HybridClock initialized: now 1790246102416925 us; error 35 us; skew 500 ppm
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246102, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:02.419927 5476 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:35:02.420766 5476 webserver.cc:460] Webserver started at http://127.2.193.130:43159/ using document root <none> and password file <none>
I20260924 10:35:02.421375 5476 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:02.421437 5476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:02.421607 5476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:02.423660 5476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "82b0129fb00549b78718c8dd6db03309"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "7c3b530e1e2f9cb32e67fa4b21f9f994"
server_key_iv: "dc1f84f588001a905b5596ae6f571306"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.424193 5476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "82b0129fb00549b78718c8dd6db03309"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "7c3b530e1e2f9cb32e67fa4b21f9f994"
server_key_iv: "dc1f84f588001a905b5596ae6f571306"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.427917 5476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:02.430540 5491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.431999 5476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:35:02.432157 5476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "82b0129fb00549b78718c8dd6db03309"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "7c3b530e1e2f9cb32e67fa4b21f9f994"
server_key_iv: "dc1f84f588001a905b5596ae6f571306"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.432282 5476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:02.446008 5476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:02.448982 5476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:02.449258 5476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:02.449869 5476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:02.450748 5476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:02.450822 5476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.450904 5476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:02.450955 5476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.459906 5476 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:32861
I20260924 10:35:02.459921 5604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:32861 every 8 connection(s)
I20260924 10:35:02.460837 5476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:02.462684 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 5476
I20260924 10:35:02.462790 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:02.463069 2822 external_mini_cluster.cc:1428] Setting key 561179243405b699044dd0610bd3d3be
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246102, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
I20260924 10:35:02.476119 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:02.462769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:58665 (local address 127.2.193.190:35297)
0924 10:35:02.462910 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:02.462919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:02.463974 (+ 1055us) server_negotiation.cc:407] Connection header received
0924 10:35:02.465874 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:02.465877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:02.465922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:02.465994 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:02.467325 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.467874 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.468994 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.469363 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.469479 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:02.472230 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:02.472233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:02.472238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:02.472267 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:02.473753 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:02.474418 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:02.474422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:02.474424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:02.474507 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:02.474847 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:02.474852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:02.474853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:02.475108 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:02.475221 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:02.475816 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 10:35:02.475950 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:35:02.477005 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:02.463287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:35297 (local address 127.2.193.130:58665)
0924 10:35:02.463815 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:02.463870 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:02.465639 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:02.466182 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:02.466193 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:02.466665 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:02.467162 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:02.467176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.467982 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.467984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:02.468862 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:02.468870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.469532 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.469536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:02.469725 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:02.470349 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:02.470365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:02.472051 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:02.473902 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:02.473908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:02.473926 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:02.474311 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:02.474599 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:02.474604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:02.474605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:02.474754 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:02.475218 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:02.475226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:02.475560 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 10:35:02.475776 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 10:35:02.476131 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":168,"threads_started":1}
I20260924 10:35:02.478406 5605 heartbeater.cc:347] Connected to a master server at 127.2.193.190:35297
I20260924 10:35:02.478675 5605 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:02.479242 5605 heartbeater.cc:511] Master 127.2.193.190:35297 requested a full tablet report, sending...
I20260924 10:35:02.480371 5280 ts_manager.cc:194] Registered new tserver with Master: 82b0129fb00549b78718c8dd6db03309 (127.2.193.130:32861)
I20260924 10:35:02.480973 5280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:58665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:35:02.491869 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:35297
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:02.604666 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:02.604948 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:02.604992 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:02.608964 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:02.609023 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:02.609086 5610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:02.614472 5610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:35297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:02.615445 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:02.616432 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:02.625850 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:02.625949 5610 server_base.cc:1034] running on GCE node
W20260924 10:35:02.625978 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:02.625869 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:02.626511 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:02.627139 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:02.628319 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790246102628309 us; error 26 us; skew 500 ppm
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246102, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:02.631222 5610 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:35:02.632058 5610 webserver.cc:460] Webserver started at http://127.2.193.131:39705/ using document root <none> and password file <none>
I20260924 10:35:02.632563 5610 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:02.632620 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:02.632803 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:02.634825 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "476dee2aa939483f9583889c8f76ceab"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "8fdb0c4528e1eae715f72e46538d4d09"
server_key_iv: "910d7f3f9390342b1a11427bdfd5b78e"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.635385 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "476dee2aa939483f9583889c8f76ceab"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "8fdb0c4528e1eae715f72e46538d4d09"
server_key_iv: "910d7f3f9390342b1a11427bdfd5b78e"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.639065 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:35:02.641463 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.642655 5610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:35:02.642778 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "476dee2aa939483f9583889c8f76ceab"
format_stamp: "Formatted at 2026-09-24 10:35:02 on dist-test-slave-8wc7"
server_key: "8fdb0c4528e1eae715f72e46538d4d09"
server_key_iv: "910d7f3f9390342b1a11427bdfd5b78e"
server_key_version: "encryptionkey@0"
I20260924 10:35:02.642887 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:02.656997 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:02.659873 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:02.660075 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:02.660665 5610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:02.661557 5610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:02.661604 5610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.661674 5610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:02.661707 5610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:02.670553 5610 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:43211
I20260924 10:35:02.670573 5738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:43211 every 8 connection(s)
I20260924 10:35:02.671432 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:02.678334 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 5610
I20260924 10:35:02.678428 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:02.678781 2822 external_mini_cluster.cc:1428] Setting key a5f1266f02cbc0cd3fdd046c79a76723
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246102, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:02.685142 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:02.673327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:44541 (local address 127.2.193.190:35297)
0924 10:35:02.673473 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:02.673476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:02.674161 (+ 685us) server_negotiation.cc:407] Connection header received
0924 10:35:02.675488 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:02.675491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:02.675548 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:02.675654 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:02.676815 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.677436 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.678679 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.678965 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.679077 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:02.681912 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:02.681916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:02.681918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:02.681944 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:02.683164 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:02.683660 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:02.683665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:02.683667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:02.683751 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:02.684078 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:02.684083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:02.684085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:02.684266 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:02.684369 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:02.684858 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 10:35:02.684983 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:35:02.685966 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:02.673605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:35297 (local address 127.2.193.131:44541)
0924 10:35:02.674012 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:02.674052 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:02.675272 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:02.675822 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:02.675830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:02.676193 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:02.676669 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:02.676681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.677573 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.677577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:02.678556 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:02.678567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:02.679125 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:02.679129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:02.679341 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:02.680038 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:02.680061 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:02.681747 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:02.683282 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:02.683287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:02.683302 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:02.683558 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:02.683840 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:02.683845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:02.683847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:02.683985 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:02.684376 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:02.684384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:02.684632 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 10:35:02.684819 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 10:35:02.685135 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260924 10:35:02.687099 5739 heartbeater.cc:347] Connected to a master server at 127.2.193.190:35297
I20260924 10:35:02.687336 5739 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:02.687841 5739 heartbeater.cc:511] Master 127.2.193.190:35297 requested a full tablet report, sending...
I20260924 10:35:02.688805 5280 ts_manager.cc:194] Registered new tserver with Master: 476dee2aa939483f9583889c8f76ceab (127.2.193.131:43211)
I20260924 10:35:02.689411 5280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:44541
I20260924 10:35:02.692488 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:35:02.705183 5319 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:02.703789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.2.193.190:35297)
0924 10:35:02.703934 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:02.703938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:02.704102 (+ 164us) server_negotiation.cc:407] Connection header received
0924 10:35:02.704218 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:02.704220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:02.704262 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:02.704341 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:02.705020 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34878: BlockingRecv error: recv got EOF from 127.0.0.1:34878 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:35:02.706182 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 5342
I20260924 10:35:02.711561 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 5476
I20260924 10:35:02.716776 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 5610
I20260924 10:35:02.722163 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 5248
2026-09-24T10:35:02Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2149 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5751](info): set up 2 sockets
Sep 24 10:35:02 dist-test-slave-8wc7 krb5kdc[5751](info): commencing operation
krb5kdc: starting...
W20260924 10:35:03.757073 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:03 dist-test-slave-8wc7 krb5kdc[5751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:03.880934 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46277
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:03.993955 5764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:03.994246 5764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:03.994287 5764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:03.998605 5764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:03.998677 5764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:03.998696 5764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:03.998713 5764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:03.998728 5764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:04.004231 5764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:04.005396 5764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:04.006528 5764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:04.015178 5772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:04.015225 5764 server_base.cc:1034] running on GCE node
W20260924 10:35:04.015173 5770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:04.015173 5769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:04.016038 5764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:04.017230 5764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:04.018426 5764 hybrid_clock.cc:648] HybridClock initialized: now 1790246104018396 us; error 46 us; skew 500 ppm
Sep 24 10:35:04 dist-test-slave-8wc7 krb5kdc[5751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246104, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:04.025547 5764 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:04.026502 5764 webserver.cc:460] Webserver started at http://127.2.193.190:40975/ using document root <none> and password file <none>
I20260924 10:35:04.027086 5764 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:04.027176 5764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:04.027431 5764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:04.029603 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "4917ea6bfda3490eb06c2f594ac5071f"
format_stamp: "Formatted at 2026-09-24 10:35:04 on dist-test-slave-8wc7"
server_key: "d7cbf684ae9ab1f1dcc8a120a163ca34"
server_key_iv: "515109365aac8f320639bd7e9a2d7829"
server_key_version: "encryptionkey@0"
I20260924 10:35:04.030193 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "4917ea6bfda3490eb06c2f594ac5071f"
format_stamp: "Formatted at 2026-09-24 10:35:04 on dist-test-slave-8wc7"
server_key: "d7cbf684ae9ab1f1dcc8a120a163ca34"
server_key_iv: "515109365aac8f320639bd7e9a2d7829"
server_key_version: "encryptionkey@0"
I20260924 10:35:04.034003 5764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:04.036446 5779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:04.037839 5764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:04.037998 5764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "4917ea6bfda3490eb06c2f594ac5071f"
format_stamp: "Formatted at 2026-09-24 10:35:04 on dist-test-slave-8wc7"
server_key: "d7cbf684ae9ab1f1dcc8a120a163ca34"
server_key_iv: "515109365aac8f320639bd7e9a2d7829"
server_key_version: "encryptionkey@0"
I20260924 10:35:04.038115 5764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:04.062958 5764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:04.066066 5764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:04.066289 5764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:04.073581 5764 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:46277
I20260924 10:35:04.073899 5831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:46277 every 8 connection(s)
I20260924 10:35:04.074648 5764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:04.077574 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:04.077813 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 5764
I20260924 10:35:04.077946 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:04.078230 2822 external_mini_cluster.cc:1428] Setting key fde1dcae84b09bdbf6e28b0a8b49e01e
I20260924 10:35:04.084865 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f: Bootstrap starting.
I20260924 10:35:04.087604 5832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f: Neither blocks nor log segments found. Creating new log.
Sep 24 10:35:04 dist-test-slave-8wc7 krb5kdc[5751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:04.088464 5832 log.cc:824] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:04.090977 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f: No bootstrap required, opened a new log
I20260924 10:35:04.093688 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:04.079603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.2.193.190:46277)
0924 10:35:04.080042 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:04.080077 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:04.080121 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:35:04.080990 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:04.081015 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:04.081481 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:04.081888 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:04.082608 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:04.084110 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:04.084993 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:04.085675 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:04.085843 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:04.088539 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:04.088547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:04.088566 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:04.088599 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:04.090678 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:04.091180 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:04.091192 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:04.091196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:04.091315 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:04.091635 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:04.091639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:04.091641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:04.092057 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:04.092266 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:04.092567 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 10:35:04.092864 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260924 10:35:04.095034 5832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4917ea6bfda3490eb06c2f594ac5071f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46277 } }
I20260924 10:35:04.095295 5832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:04.095371 5832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4917ea6bfda3490eb06c2f594ac5071f, State: Initialized, Role: FOLLOWER
I20260924 10:35:04.095863 5832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4917ea6bfda3490eb06c2f594ac5071f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46277 } }
I20260924 10:35:04.096017 5832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:04.096091 5832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:04.096194 5832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:04.096858 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:35:04.097455 5832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4917ea6bfda3490eb06c2f594ac5071f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46277 } }
I20260924 10:35:04.097803 5832 leader_election.cc:302] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4917ea6bfda3490eb06c2f594ac5071f; no voters:
I20260924 10:35:04.098143 5832 leader_election.cc:288] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:04.098227 5837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:04.098440 5837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [term 1 LEADER]: Becoming Leader. State: Replica: 4917ea6bfda3490eb06c2f594ac5071f, State: Running, Role: LEADER
I20260924 10:35:04.098811 5837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4917ea6bfda3490eb06c2f594ac5071f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46277 } }
I20260924 10:35:04.099512 5832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:04.100037 5837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4917ea6bfda3490eb06c2f594ac5071f. Latest consensus state: current_term: 1 leader_uuid: "4917ea6bfda3490eb06c2f594ac5071f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4917ea6bfda3490eb06c2f594ac5071f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46277 } } }
I20260924 10:35:04.100234 5837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:04.100436 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4917ea6bfda3490eb06c2f594ac5071f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4917ea6bfda3490eb06c2f594ac5071f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46277 } } }
I20260924 10:35:04.100598 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:04.100679 5844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:04.104297 5844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:04.111101 5844 catalog_manager.cc:1347] Generated new cluster ID: 75e76636d62f4cbab3b437c762e14fc7
I20260924 10:35:04.111179 5844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:04.120697 5844 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:04.121313 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:04.114795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.2.193.190:46277)
0924 10:35:04.114920 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:04.114925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:04.115166 (+ 241us) server_negotiation.cc:407] Connection header received
0924 10:35:04.115278 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:04.115281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:04.115333 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:04.115446 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:04.116022 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:04.116849 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:04.118068 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:04.118466 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:04.118578 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:04.120152 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:04.120157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:04.120160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:04.120793 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:04.120829 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:04.120848 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:04.120944 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:04.121075 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:35:04.121140 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:35:04.122063 5844 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:04.133127 5844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4917ea6bfda3490eb06c2f594ac5071f: Generated new TSK 0
I20260924 10:35:04.134233 5844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:35:04.144979 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 5764
2026-09-24T10:35:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1420 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:04 dist-test-slave-8wc7 krb5kdc[5865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:04 dist-test-slave-8wc7 krb5kdc[5865](info): set up 2 sockets
Sep 24 10:35:04 dist-test-slave-8wc7 krb5kdc[5865](info): commencing operation
krb5kdc: starting...
W20260924 10:35:05.185386 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:05 dist-test-slave-8wc7 krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:05.311717 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:41923
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:41923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:05.426426 5878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:05.426708 5878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:05.426749 5878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:05.430797 5878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:05.430863 5878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:05.430883 5878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:05.430900 5878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:05.430917 5878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:05.436196 5878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:41923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:41923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:05.437294 5878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:05.438297 5878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:05.447438 5883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:05.447477 5884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:05.447711 5878 server_base.cc:1034] running on GCE node
W20260924 10:35:05.447438 5886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:05.448525 5878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:05.449873 5878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:05.451078 5878 hybrid_clock.cc:648] HybridClock initialized: now 1790246105451062 us; error 34 us; skew 500 ppm
Sep 24 10:35:05 dist-test-slave-8wc7 krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:05.454005 5878 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:05.455020 5878 webserver.cc:460] Webserver started at http://127.2.193.190:43365/ using document root <none> and password file <none>
I20260924 10:35:05.455627 5878 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:05.455724 5878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:05.455969 5878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:05.458210 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "ad49f66e08564019b336f694ac1e8428"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "bbd9819ee4816e1c39351013e369c4fa"
server_key_iv: "a7347820c3d3776b71d5ec2816766a5b"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.458813 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "ad49f66e08564019b336f694ac1e8428"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "bbd9819ee4816e1c39351013e369c4fa"
server_key_iv: "a7347820c3d3776b71d5ec2816766a5b"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.462745 5878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:05.465544 5893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:05.466823 5878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:35:05.466990 5878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "ad49f66e08564019b336f694ac1e8428"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "bbd9819ee4816e1c39351013e369c4fa"
server_key_iv: "a7347820c3d3776b71d5ec2816766a5b"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.467119 5878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:05.479024 5878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:05.482234 5878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:05.482455 5878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:05.489746 5878 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:41923
I20260924 10:35:05.490034 5945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:41923 every 8 connection(s)
I20260924 10:35:05.490789 5878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:05.493734 5946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:05.498535 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 5878
I20260924 10:35:05.498673 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:05.498977 2822 external_mini_cluster.cc:1428] Setting key 91f3abb4ceab4436131f3a39c943eed0
I20260924 10:35:05.499732 5946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428: Bootstrap starting.
I20260924 10:35:05.502271 5946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:05.503244 5946 log.cc:824] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:05.505518 5946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428: No bootstrap required, opened a new log
Sep 24 10:35:05 dist-test-slave-8wc7 krb5kdc[5865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:05.510157 5946 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad49f66e08564019b336f694ac1e8428" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 41923 } }
I20260924 10:35:05.510450 5946 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:05.510545 5946 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad49f66e08564019b336f694ac1e8428, State: Initialized, Role: FOLLOWER
I20260924 10:35:05.511015 5946 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad49f66e08564019b336f694ac1e8428" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 41923 } }
I20260924 10:35:05.511170 5946 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:05.511258 5946 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:05.511368 5946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:05.512331 5946 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad49f66e08564019b336f694ac1e8428" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 41923 } }
I20260924 10:35:05.512706 5946 leader_election.cc:302] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad49f66e08564019b336f694ac1e8428; no voters:
I20260924 10:35:05.513098 5946 leader_election.cc:288] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:05.513196 5951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:05.513401 5951 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [term 1 LEADER]: Becoming Leader. State: Replica: ad49f66e08564019b336f694ac1e8428, State: Running, Role: LEADER
I20260924 10:35:05.513792 5951 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad49f66e08564019b336f694ac1e8428" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 41923 } }
I20260924 10:35:05.514160 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:05.500388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.2.193.190:41923)
0924 10:35:05.500818 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:05.500834 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:05.500871 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:35:05.501585 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:05.501603 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:05.501945 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:05.502241 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:05.503183 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:05.504213 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:05.505385 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:05.505908 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:05.506038 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:05.508830 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:05.508835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:05.508850 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:05.508877 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:05.510813 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:05.511290 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:05.511296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:05.511302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:05.511505 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:05.511801 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:05.511806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:05.511808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:05.512224 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:05.512429 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:05.512738 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 10:35:05.513072 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":152,"threads_started":1}
I20260924 10:35:05.514500 5946 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:05.515460 5953 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad49f66e08564019b336f694ac1e8428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad49f66e08564019b336f694ac1e8428" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 41923 } } }
I20260924 10:35:05.515596 5953 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:05.515918 5952 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad49f66e08564019b336f694ac1e8428. Latest consensus state: current_term: 1 leader_uuid: "ad49f66e08564019b336f694ac1e8428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad49f66e08564019b336f694ac1e8428" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 41923 } } }
I20260924 10:35:05.516008 5952 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:05.516836 5960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:05.519714 5960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:05.525995 5960 catalog_manager.cc:1347] Generated new cluster ID: 15f9a6ed2ec841ba89aea66759223491
I20260924 10:35:05.526094 5960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:05.547724 5960 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:05.548758 5960 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:05.558120 5960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad49f66e08564019b336f694ac1e8428: Generated new TSK 0
I20260924 10:35:05.558888 5960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:35:05.593120 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:41923
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:05.707677 5972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:05.707973 5972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:05.708017 5972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:05.712275 5972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:05.712347 5972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:05.712471 5972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:05.718273 5972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:41923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:05.719462 5972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:05.720569 5972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:05.730603 5980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:05.730633 5978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:05.730801 5972 server_base.cc:1034] running on GCE node
W20260924 10:35:05.730603 5977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:05.731415 5972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:05.732120 5972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:05.733263 5972 hybrid_clock.cc:648] HybridClock initialized: now 1790246105733212 us; error 57 us; skew 500 ppm
Sep 24 10:35:05 dist-test-slave-8wc7 krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:05.736151 5972 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:35:05.737082 5972 webserver.cc:460] Webserver started at http://127.2.193.129:40629/ using document root <none> and password file <none>
I20260924 10:35:05.737692 5972 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:05.737787 5972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:05.738034 5972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:05.740170 5972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "0cf650f93e2841569d5a4c0f7854d863"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "5155a0922c0049e719e7dbecddad71a3"
server_key_iv: "758be712e19ca4d798368c4ee6da116e"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.740767 5972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "0cf650f93e2841569d5a4c0f7854d863"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "5155a0922c0049e719e7dbecddad71a3"
server_key_iv: "758be712e19ca4d798368c4ee6da116e"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.744673 5972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 10:35:05.747251 5987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:05.748588 5972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:05.748730 5972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "0cf650f93e2841569d5a4c0f7854d863"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "5155a0922c0049e719e7dbecddad71a3"
server_key_iv: "758be712e19ca4d798368c4ee6da116e"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.748850 5972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:05.772918 5972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:05.776104 5972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:05.776326 5972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:05.776966 5972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:05.777930 5972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:05.778012 5972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:05.778086 5972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:05.778129 5972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:05.786983 5972 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:37501
I20260924 10:35:05.787000 6100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:37501 every 8 connection(s)
I20260924 10:35:05.787969 5972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:05.789919 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 5972
I20260924 10:35:05.790037 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:05.790354 2822 external_mini_cluster.cc:1428] Setting key 7b7f8ab8062a63cd33cdf1c6f7875b89
Sep 24 10:35:05 dist-test-slave-8wc7 krb5kdc[5865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:05.802109 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:05.789810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:36227 (local address 127.2.193.190:41923)
0924 10:35:05.789947 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:05.789951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:05.791084 (+ 1133us) server_negotiation.cc:407] Connection header received
0924 10:35:05.792589 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:05.792592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:05.792644 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:05.792720 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:05.793961 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:05.794493 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:05.795638 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:05.795875 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:05.795998 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:05.798525 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:05.798528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:05.798530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:05.798553 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:05.800035 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:05.800549 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:05.800554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:05.800556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:05.800635 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:05.800966 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:05.800970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:05.800972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:05.801154 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:05.801312 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:05.801817 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 10:35:05.801940 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:35:05.802976 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:05.790362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:41923 (local address 127.2.193.129:36227)
0924 10:35:05.790935 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:05.790995 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:05.792374 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:05.792888 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:05.792896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:05.793328 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:05.793818 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:05.793831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:05.794624 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:05.794634 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:05.795511 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:05.795520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:05.796030 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:05.796033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:05.796204 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:05.796809 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:05.796826 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:05.798339 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:05.800168 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:05.800174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:05.800190 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:05.800447 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:05.800726 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:05.800731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:05.800733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:05.800873 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:05.801309 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:05.801316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:05.801579 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 10:35:05.801766 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 10:35:05.802105 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":157,"threads_started":1}
I20260924 10:35:05.804517 6101 heartbeater.cc:347] Connected to a master server at 127.2.193.190:41923
I20260924 10:35:05.804771 6101 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:05.805333 6101 heartbeater.cc:511] Master 127.2.193.190:41923 requested a full tablet report, sending...
I20260924 10:35:05.807116 5910 ts_manager.cc:194] Registered new tserver with Master: 0cf650f93e2841569d5a4c0f7854d863 (127.2.193.129:37501)
I20260924 10:35:05.808843 5910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:36227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:35:05.820358 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:41923
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:05.932761 6106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:05.933107 6106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:05.933185 6106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:05.937407 6106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:05.937508 6106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:05.937604 6106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:05.943167 6106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:41923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:05.944335 6106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:05.945497 6106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:05.955621 6111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:05.955621 6112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:05.955665 6114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:05.956059 6106 server_base.cc:1034] running on GCE node
I20260924 10:35:05.956552 6106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:05.957345 6106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:05.958509 6106 hybrid_clock.cc:648] HybridClock initialized: now 1790246105958490 us; error 32 us; skew 500 ppm
Sep 24 10:35:05 dist-test-slave-8wc7 krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:05.961386 6106 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:35:05.962312 6106 webserver.cc:460] Webserver started at http://127.2.193.130:41379/ using document root <none> and password file <none>
I20260924 10:35:05.962906 6106 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:05.962980 6106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:05.963230 6106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:05.965480 6106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "6102e0e36f4a4d998499ec583c480254"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "690c6ba9f66ded8cddd3bd00e8ad6a98"
server_key_iv: "2cb34d237aa16c8ee72634e393a7c932"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.966152 6106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "6102e0e36f4a4d998499ec583c480254"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "690c6ba9f66ded8cddd3bd00e8ad6a98"
server_key_iv: "2cb34d237aa16c8ee72634e393a7c932"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.970157 6106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:05.972687 6121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:05.973987 6106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:05.974120 6106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "6102e0e36f4a4d998499ec583c480254"
format_stamp: "Formatted at 2026-09-24 10:35:05 on dist-test-slave-8wc7"
server_key: "690c6ba9f66ded8cddd3bd00e8ad6a98"
server_key_iv: "2cb34d237aa16c8ee72634e393a7c932"
server_key_version: "encryptionkey@0"
I20260924 10:35:05.974236 6106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:05.995112 6106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:05.998293 6106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:05.998520 6106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:05.999161 6106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:06.000033 6106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:06.000080 6106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:06.000152 6106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:06.000188 6106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:06.009220 6106 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:39099
I20260924 10:35:06.009256 6234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:39099 every 8 connection(s)
I20260924 10:35:06.010176 6106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:06.017102 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 6106
I20260924 10:35:06.017294 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:06.017622 2822 external_mini_cluster.cc:1428] Setting key 43264183dc47c7a6f7f9972ac28740b2
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[5865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246105, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:06.025092 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:06.012053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:36813 (local address 127.2.193.190:41923)
0924 10:35:06.012205 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:06.012209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:06.013101 (+ 892us) server_negotiation.cc:407] Connection header received
0924 10:35:06.014604 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:06.014608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:06.014667 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:06.014774 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:06.016090 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.016622 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.018001 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.018230 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.018346 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:06.021042 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:06.021046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:06.021048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:06.021074 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:06.022630 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:06.023316 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:06.023319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:06.023320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:06.023399 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:06.023824 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:06.023826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:06.023827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:06.024032 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:06.024110 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:06.024844 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 10:35:06.024965 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:35:06.026413 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:06.012473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:41923 (local address 127.2.193.130:36813)
0924 10:35:06.012929 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:06.012983 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:06.014370 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:06.014992 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:06.014999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:06.015373 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:06.015928 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:06.015953 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.016757 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.016760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:06.017868 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:06.017878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.018388 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.018392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:06.018567 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:06.019185 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:06.019201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:06.020844 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:06.022760 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:06.022769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:06.022791 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:06.023206 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:06.023502 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:06.023505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:06.023507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:06.023717 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:06.024144 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:06.024153 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:06.024557 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 10:35:06.024814 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 10:35:06.025238 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:06.027828 6235 heartbeater.cc:347] Connected to a master server at 127.2.193.190:41923
I20260924 10:35:06.028097 6235 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:06.028582 6235 heartbeater.cc:511] Master 127.2.193.190:41923 requested a full tablet report, sending...
I20260924 10:35:06.029735 5910 ts_manager.cc:194] Registered new tserver with Master: 6102e0e36f4a4d998499ec583c480254 (127.2.193.130:39099)
I20260924 10:35:06.030412 5910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:36813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:35:06.047755 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:41923
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:06.163982 6240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:06.164266 6240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:06.164310 6240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:06.168550 6240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:06.168630 6240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:06.168692 6240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:06.174446 6240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:41923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:06.175554 6240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:06.176589 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:06.186867 6245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:06.186895 6246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:06.186956 6240 server_base.cc:1034] running on GCE node
W20260924 10:35:06.186867 6248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:06.187687 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:06.188369 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:06.189553 6240 hybrid_clock.cc:648] HybridClock initialized: now 1790246106189533 us; error 33 us; skew 500 ppm
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246106, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:06.192427 6240 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:35:06.193451 6240 webserver.cc:460] Webserver started at http://127.2.193.131:34353/ using document root <none> and password file <none>
I20260924 10:35:06.194051 6240 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:06.194144 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:06.194370 6240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:06.196493 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "a46ec357336d48739d2b11fa9e62944c"
format_stamp: "Formatted at 2026-09-24 10:35:06 on dist-test-slave-8wc7"
server_key: "48f4ffdd059a42ca760c9dfffbc321ef"
server_key_iv: "5b124b4da5aa5cef88005f0adce558d7"
server_key_version: "encryptionkey@0"
I20260924 10:35:06.197077 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "a46ec357336d48739d2b11fa9e62944c"
format_stamp: "Formatted at 2026-09-24 10:35:06 on dist-test-slave-8wc7"
server_key: "48f4ffdd059a42ca760c9dfffbc321ef"
server_key_iv: "5b124b4da5aa5cef88005f0adce558d7"
server_key_version: "encryptionkey@0"
I20260924 10:35:06.200855 6240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:06.203471 6255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:06.204779 6240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:06.204917 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "a46ec357336d48739d2b11fa9e62944c"
format_stamp: "Formatted at 2026-09-24 10:35:06 on dist-test-slave-8wc7"
server_key: "48f4ffdd059a42ca760c9dfffbc321ef"
server_key_iv: "5b124b4da5aa5cef88005f0adce558d7"
server_key_version: "encryptionkey@0"
I20260924 10:35:06.205040 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:06.231261 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:06.234824 6240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:06.235049 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:06.235661 6240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:06.236500 6240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:06.236567 6240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:06.236639 6240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:06.236683 6240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:06.247162 6240 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:40147
I20260924 10:35:06.247203 6368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:40147 every 8 connection(s)
I20260924 10:35:06.248401 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:06.254820 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 6240
I20260924 10:35:06.254971 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:06.255268 2822 external_mini_cluster.cc:1428] Setting key 62ded5f72fb068e05c26b7d5d1e90bc5
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[5865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246106, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:06.262965 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:06.250486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:40267 (local address 127.2.193.190:41923)
0924 10:35:06.250625 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:06.250629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:06.251429 (+ 800us) server_negotiation.cc:407] Connection header received
0924 10:35:06.252937 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:06.252940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:06.252996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:06.253077 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:06.254289 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.254839 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.256330 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.256590 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.256690 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:06.259371 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:06.259375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:06.259378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:06.259408 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:06.260855 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:06.261459 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:06.261463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:06.261464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:06.261549 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:06.261892 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:06.261896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:06.261898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:06.262080 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:06.262182 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:06.262680 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 10:35:06.262803 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:35:06.263854 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:06.250830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:41923 (local address 127.2.193.131:40267)
0924 10:35:06.251278 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:06.251322 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:06.252723 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:06.253285 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:06.253292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:06.253659 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:06.254144 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:06.254157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.255017 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.255021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:06.256216 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:06.256228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.256774 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.256778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:06.256980 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:06.257697 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:06.257714 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:06.259202 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:06.260995 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:06.261001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:06.261016 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:06.261358 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:06.261645 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:06.261648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:06.261650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:06.261791 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:06.262180 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:06.262187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:06.262443 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 10:35:06.262629 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:35:06.262958 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
I20260924 10:35:06.265014 6369 heartbeater.cc:347] Connected to a master server at 127.2.193.190:41923
I20260924 10:35:06.265302 6369 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:06.265797 6369 heartbeater.cc:511] Master 127.2.193.190:41923 requested a full tablet report, sending...
I20260924 10:35:06.266769 5910 ts_manager.cc:194] Registered new tserver with Master: a46ec357336d48739d2b11fa9e62944c (127.2.193.131:40147)
I20260924 10:35:06.267326 5910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:40267
I20260924 10:35:06.269232 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[5865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246106, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:06.294790 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:06.286348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.2.193.190:41923)
0924 10:35:06.286537 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:06.286542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:06.286681 (+ 139us) server_negotiation.cc:407] Connection header received
0924 10:35:06.286862 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:06.286865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:06.286907 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:06.287002 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:06.287750 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.288273 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.289234 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:06.289463 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:06.289575 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:06.292035 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:06.292039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:06.292041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:06.292068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:06.293355 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:06.293740 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:06.293743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:06.293744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:06.293817 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:06.294046 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:06.294048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:06.294049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:06.294268 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:06.294371 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:06.294567 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:35:06.294666 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 10:35:06.295212 5910 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55144
I20260924 10:35:06.296106 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 5972
I20260924 10:35:06.301683 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 6106
I20260924 10:35:06.306918 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 6240
I20260924 10:35:06.312088 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 5878
2026-09-24T10:35:06Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2169 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[6381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[6381](info): set up 2 sockets
Sep 24 10:35:06 dist-test-slave-8wc7 krb5kdc[6381](info): commencing operation
krb5kdc: starting...
W20260924 10:35:07.349403 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:07 dist-test-slave-8wc7 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:07.477002 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:39261
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:07.590055 6394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:07.590402 6394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:07.590499 6394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:07.594789 6394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:07.594895 6394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:07.594935 6394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:07.594992 6394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:07.595033 6394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:07.600688 6394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:39261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:07.601954 6394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:07.603070 6394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:07.612046 6400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:07.612172 6394 server_base.cc:1034] running on GCE node
W20260924 10:35:07.612059 6399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:07.612039 6402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:07.612995 6394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:07.614276 6394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:07.615475 6394 hybrid_clock.cc:648] HybridClock initialized: now 1790246107615441 us; error 48 us; skew 500 ppm
Sep 24 10:35:07 dist-test-slave-8wc7 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246107, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:07.618436 6394 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:07.619436 6394 webserver.cc:460] Webserver started at http://127.2.193.190:39679/ using document root <none> and password file <none>
I20260924 10:35:07.620069 6394 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:07.620170 6394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:07.620421 6394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:07.622710 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "91880a32ee6840a588ba57430151d0b1"
format_stamp: "Formatted at 2026-09-24 10:35:07 on dist-test-slave-8wc7"
server_key: "b30656f60e9c77863af43f78c5723247"
server_key_iv: "3ce9bf0173b274ad07c029a97290dce9"
server_key_version: "encryptionkey@0"
I20260924 10:35:07.623332 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "91880a32ee6840a588ba57430151d0b1"
format_stamp: "Formatted at 2026-09-24 10:35:07 on dist-test-slave-8wc7"
server_key: "b30656f60e9c77863af43f78c5723247"
server_key_iv: "3ce9bf0173b274ad07c029a97290dce9"
server_key_version: "encryptionkey@0"
I20260924 10:35:07.627322 6394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:07.629884 6409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:07.631249 6394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:07.631405 6394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "91880a32ee6840a588ba57430151d0b1"
format_stamp: "Formatted at 2026-09-24 10:35:07 on dist-test-slave-8wc7"
server_key: "b30656f60e9c77863af43f78c5723247"
server_key_iv: "3ce9bf0173b274ad07c029a97290dce9"
server_key_version: "encryptionkey@0"
I20260924 10:35:07.631531 6394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:07.644483 6394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:07.648057 6394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:07.648280 6394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:07.655687 6394 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:39261
I20260924 10:35:07.655704 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:39261 every 8 connection(s)
I20260924 10:35:07.656721 6394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:07.659526 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:07.663367 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 6394
I20260924 10:35:07.663491 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:07.663806 2822 external_mini_cluster.cc:1428] Setting key 992c7cdc24b65dac10de1552ef58186d
I20260924 10:35:07.665782 6462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1: Bootstrap starting.
I20260924 10:35:07.668063 6462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:07.668829 6462 log.cc:824] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:07.670765 6462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1: No bootstrap required, opened a new log
Sep 24 10:35:07 dist-test-slave-8wc7 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:07.674729 6462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91880a32ee6840a588ba57430151d0b1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39261 } }
I20260924 10:35:07.674971 6462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:07.675032 6462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91880a32ee6840a588ba57430151d0b1, State: Initialized, Role: FOLLOWER
I20260924 10:35:07.675489 6462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91880a32ee6840a588ba57430151d0b1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39261 } }
I20260924 10:35:07.675611 6462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:07.675711 6462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:07.675819 6462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:07.676783 6462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91880a32ee6840a588ba57430151d0b1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39261 } }
I20260924 10:35:07.677153 6462 leader_election.cc:302] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91880a32ee6840a588ba57430151d0b1; no voters:
I20260924 10:35:07.677593 6462 leader_election.cc:288] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:07.677740 6467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:07.677989 6467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [term 1 LEADER]: Becoming Leader. State: Replica: 91880a32ee6840a588ba57430151d0b1, State: Running, Role: LEADER
I20260924 10:35:07.678344 6467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91880a32ee6840a588ba57430151d0b1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39261 } }
I20260924 10:35:07.679009 6462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:07.679018 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:07.665317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.2.193.190:39261)
0924 10:35:07.665753 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:07.665768 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:07.665805 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:35:07.666522 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:07.666541 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:07.666886 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:07.667243 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:07.668076 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:07.669243 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:07.670111 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:07.670599 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:07.670776 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:07.673574 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:07.673580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:07.673595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:07.673624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:07.675600 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:07.676155 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:07.676161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:07.676168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:07.676301 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:07.676594 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:07.676598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:07.676600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:07.677114 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:07.677370 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:07.677685 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 10:35:07.678038 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":142,"threads_started":1}
I20260924 10:35:07.680058 6468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91880a32ee6840a588ba57430151d0b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91880a32ee6840a588ba57430151d0b1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39261 } } }
I20260924 10:35:07.680204 6468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:07.680552 6469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91880a32ee6840a588ba57430151d0b1. Latest consensus state: current_term: 1 leader_uuid: "91880a32ee6840a588ba57430151d0b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91880a32ee6840a588ba57430151d0b1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39261 } } }
I20260924 10:35:07.680644 6469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:07.681434 6476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:07.685109 6476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:07.691490 6476 catalog_manager.cc:1347] Generated new cluster ID: 4b76ab1df9c04db89e3efc8698e58d7a
I20260924 10:35:07.691589 6476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:07.707805 6476 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:07.708940 6476 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:07.724771 6476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91880a32ee6840a588ba57430151d0b1: Generated new TSK 0
I20260924 10:35:07.725692 6476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:35:07.756671 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39261
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:07.875630 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:07.876108 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:07.876205 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:07.880607 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:07.880684 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:07.880750 6488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:07.886425 6488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:07.887665 6488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:07.888774 6488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:07.898849 6496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:07.898875 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:07.898854 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:07.899243 6488 server_base.cc:1034] running on GCE node
I20260924 10:35:07.899751 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:07.900456 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:07.901623 6488 hybrid_clock.cc:648] HybridClock initialized: now 1790246107901604 us; error 31 us; skew 500 ppm
Sep 24 10:35:07 dist-test-slave-8wc7 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246107, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:07.904536 6488 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:35:07.905584 6488 webserver.cc:460] Webserver started at http://127.2.193.129:36967/ using document root <none> and password file <none>
I20260924 10:35:07.906191 6488 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:07.906286 6488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:07.906534 6488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:07.908747 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "529559a1e9744ae89ae0215211802636"
format_stamp: "Formatted at 2026-09-24 10:35:07 on dist-test-slave-8wc7"
server_key: "7d9cae73ef71a0c2f8bbf9f7417c88d4"
server_key_iv: "b2f67317a7f6513f64cf9c33334cc685"
server_key_version: "encryptionkey@0"
I20260924 10:35:07.909684 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "529559a1e9744ae89ae0215211802636"
format_stamp: "Formatted at 2026-09-24 10:35:07 on dist-test-slave-8wc7"
server_key: "7d9cae73ef71a0c2f8bbf9f7417c88d4"
server_key_iv: "b2f67317a7f6513f64cf9c33334cc685"
server_key_version: "encryptionkey@0"
I20260924 10:35:07.913793 6488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:07.916373 6503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:07.917735 6488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:07.917886 6488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "529559a1e9744ae89ae0215211802636"
format_stamp: "Formatted at 2026-09-24 10:35:07 on dist-test-slave-8wc7"
server_key: "7d9cae73ef71a0c2f8bbf9f7417c88d4"
server_key_iv: "b2f67317a7f6513f64cf9c33334cc685"
server_key_version: "encryptionkey@0"
I20260924 10:35:07.918011 6488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:07.933568 6488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:07.937107 6488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:07.937347 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:07.937963 6488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:07.938805 6488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:07.938872 6488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:07.938938 6488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:07.938985 6488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:07.947867 6488 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:45563
I20260924 10:35:07.947888 6616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:45563 every 8 connection(s)
I20260924 10:35:07.948817 6488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:07.953847 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 6488
I20260924 10:35:07.953996 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:07.954288 2822 external_mini_cluster.cc:1428] Setting key 57b68459c55b8ae8d291d3dd6b56a2fe
Sep 24 10:35:07 dist-test-slave-8wc7 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246107, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:35:07.965109 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:07.950789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:60907 (local address 127.2.193.190:39261)
0924 10:35:07.950941 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:07.950945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:07.951921 (+ 976us) server_negotiation.cc:407] Connection header received
0924 10:35:07.953431 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:07.953434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:07.953495 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:07.953596 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:07.955008 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:07.955798 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:07.957682 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:07.958104 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:07.958232 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:07.960764 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:07.960770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:07.960777 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:07.960814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:07.962723 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:07.963324 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:07.963328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:07.963330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:07.963431 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:07.963787 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:07.963790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:07.963792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:07.964072 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:07.964197 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:07.964856 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 10:35:07.964977 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:35:07.966105 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:07.951210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39261 (local address 127.2.193.129:60907)
0924 10:35:07.951666 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:07.951711 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:07.953140 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:07.953784 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:07.953791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:07.954251 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:07.954829 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:07.954849 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:07.955930 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:07.955934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:07.957505 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:07.957520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:07.958263 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:07.958266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:07.958427 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:07.959070 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:07.959087 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:07.960562 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:07.962878 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:07.962892 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:07.962907 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:07.963205 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:07.963538 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:07.963540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:07.963541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:07.963683 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:07.964230 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:07.964236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:07.964521 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 10:35:07.964780 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 10:35:07.965079 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:07.967640 6617 heartbeater.cc:347] Connected to a master server at 127.2.193.190:39261
I20260924 10:35:07.967942 6617 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:07.968482 6617 heartbeater.cc:511] Master 127.2.193.190:39261 requested a full tablet report, sending...
I20260924 10:35:07.970094 6426 ts_manager.cc:194] Registered new tserver with Master: 529559a1e9744ae89ae0215211802636 (127.2.193.129:45563)
I20260924 10:35:07.971297 6426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:60907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:35:07.983741 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39261
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:08.096566 6622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:08.096833 6622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:08.096879 6622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:08.101032 6622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:08.101106 6622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:08.101223 6622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:08.107093 6622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:08.108175 6622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:08.109244 6622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:08.119098 6627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:08.119098 6628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:08.119212 6622 server_base.cc:1034] running on GCE node
W20260924 10:35:08.119098 6630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:08.119865 6622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:08.120536 6622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:08.121722 6622 hybrid_clock.cc:648] HybridClock initialized: now 1790246108121709 us; error 26 us; skew 500 ppm
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246108, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:08.124493 6622 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:35:08.125500 6622 webserver.cc:460] Webserver started at http://127.2.193.130:38391/ using document root <none> and password file <none>
I20260924 10:35:08.126076 6622 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:08.126166 6622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:08.126403 6622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:08.128479 6622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "b8af69c741ee4cae9e612e1219a3dc9f"
format_stamp: "Formatted at 2026-09-24 10:35:08 on dist-test-slave-8wc7"
server_key: "111c6e5058c5035d8d1a0c603f0abd9a"
server_key_iv: "628a72ca38b54ac315ce2ec36d829e6f"
server_key_version: "encryptionkey@0"
I20260924 10:35:08.129038 6622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "b8af69c741ee4cae9e612e1219a3dc9f"
format_stamp: "Formatted at 2026-09-24 10:35:08 on dist-test-slave-8wc7"
server_key: "111c6e5058c5035d8d1a0c603f0abd9a"
server_key_iv: "628a72ca38b54ac315ce2ec36d829e6f"
server_key_version: "encryptionkey@0"
I20260924 10:35:08.132905 6622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 10:35:08.135600 6637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:08.136960 6622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:35:08.137066 6622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "b8af69c741ee4cae9e612e1219a3dc9f"
format_stamp: "Formatted at 2026-09-24 10:35:08 on dist-test-slave-8wc7"
server_key: "111c6e5058c5035d8d1a0c603f0abd9a"
server_key_iv: "628a72ca38b54ac315ce2ec36d829e6f"
server_key_version: "encryptionkey@0"
I20260924 10:35:08.137157 6622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:08.147406 6622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:08.150782 6622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:08.150945 6622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:08.151472 6622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:08.152246 6622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:08.152290 6622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:08.152329 6622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:08.152347 6622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:08.160966 6622 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:35293
I20260924 10:35:08.160996 6750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:35293 every 8 connection(s)
I20260924 10:35:08.161944 6622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:08.170696 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 6622
I20260924 10:35:08.170847 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:08.171115 2822 external_mini_cluster.cc:1428] Setting key 3b36447a72ef2977a730264a152097b0
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246108, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:08.177758 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:08.163859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:54841 (local address 127.2.193.190:39261)
0924 10:35:08.164014 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:08.164018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:08.164911 (+ 893us) server_negotiation.cc:407] Connection header received
0924 10:35:08.166799 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:08.166804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:08.166865 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:08.167003 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:08.168623 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.169519 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.170729 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.171050 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.171167 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:08.173740 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:08.173744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:08.173747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:08.173774 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:08.175299 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.175851 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.175854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.175856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.175941 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.176325 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.176328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.176330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.176608 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:08.176711 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:08.177516 (+ 805us) server_negotiation.cc:299] Negotiation successful
0924 10:35:08.177630 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:35:08.179157 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:08.164277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39261 (local address 127.2.193.130:54841)
0924 10:35:08.164736 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:08.164780 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:08.166510 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:08.167216 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:08.167226 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:08.167843 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:08.168440 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:08.168455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.169637 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.169639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:08.170594 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:08.170603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.171211 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.171215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:08.171410 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:08.172052 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:08.172069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:08.173561 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:08.175426 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:08.175433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:08.175451 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:08.175735 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:08.176035 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:08.176039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:08.176041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:08.176239 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:08.176746 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:08.176757 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:08.177165 (+ 408us) client_negotiation.cc:769] Sending connection context
0924 10:35:08.177458 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 10:35:08.177903 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260924 10:35:08.180581 6751 heartbeater.cc:347] Connected to a master server at 127.2.193.190:39261
I20260924 10:35:08.180863 6751 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:08.181515 6751 heartbeater.cc:511] Master 127.2.193.190:39261 requested a full tablet report, sending...
I20260924 10:35:08.182623 6426 ts_manager.cc:194] Registered new tserver with Master: b8af69c741ee4cae9e612e1219a3dc9f (127.2.193.130:35293)
I20260924 10:35:08.183219 6426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:54841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:35:08.200691 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39261
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:08.324184 6756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:08.324481 6756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:08.324527 6756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:08.328742 6756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:08.328821 6756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:08.328939 6756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:08.334679 6756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:08.335809 6756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:08.336884 6756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:08.346793 6762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:08.346807 6761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:08.346793 6764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:08.346832 6756 server_base.cc:1034] running on GCE node
I20260924 10:35:08.347470 6756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:08.348109 6756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:08.349323 6756 hybrid_clock.cc:648] HybridClock initialized: now 1790246108349300 us; error 37 us; skew 500 ppm
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246108, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:08.352092 6756 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:35:08.353101 6756 webserver.cc:460] Webserver started at http://127.2.193.131:43327/ using document root <none> and password file <none>
I20260924 10:35:08.353713 6756 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:08.353816 6756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:08.354058 6756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:08.356184 6756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "119456d5c8a3480783ef5cfd85d7eced"
format_stamp: "Formatted at 2026-09-24 10:35:08 on dist-test-slave-8wc7"
server_key: "9e6d02e4288611718a1b32863d67c3b8"
server_key_iv: "f9745e02ac566839886b2985a2c6081b"
server_key_version: "encryptionkey@0"
I20260924 10:35:08.356792 6756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "119456d5c8a3480783ef5cfd85d7eced"
format_stamp: "Formatted at 2026-09-24 10:35:08 on dist-test-slave-8wc7"
server_key: "9e6d02e4288611718a1b32863d67c3b8"
server_key_iv: "f9745e02ac566839886b2985a2c6081b"
server_key_version: "encryptionkey@0"
I20260924 10:35:08.360694 6756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:08.363245 6771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:08.364579 6756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:08.364744 6756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "119456d5c8a3480783ef5cfd85d7eced"
format_stamp: "Formatted at 2026-09-24 10:35:08 on dist-test-slave-8wc7"
server_key: "9e6d02e4288611718a1b32863d67c3b8"
server_key_iv: "f9745e02ac566839886b2985a2c6081b"
server_key_version: "encryptionkey@0"
I20260924 10:35:08.364868 6756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:08.378134 6756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:08.381431 6756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:08.381647 6756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:08.382238 6756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:08.383069 6756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:08.383140 6756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:08.383211 6756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:08.383260 6756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:08.392560 6756 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:45921
I20260924 10:35:08.392580 6884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:45921 every 8 connection(s)
I20260924 10:35:08.393544 6756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:08.401795 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 6756
I20260924 10:35:08.401954 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:08.402243 2822 external_mini_cluster.cc:1428] Setting key b44728ce02ac3b5ba03118ac174de992
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246108, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:08.407681 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:08.395444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:35531 (local address 127.2.193.190:39261)
0924 10:35:08.395587 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:08.395590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:08.396467 (+ 877us) server_negotiation.cc:407] Connection header received
0924 10:35:08.397905 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:08.397908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:08.397965 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:08.398065 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:08.399228 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.399765 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.400919 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.401146 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.401314 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:08.404174 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:08.404177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:08.404180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:08.404206 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:08.405610 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.406125 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.406130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.406132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.406213 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.406555 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.406559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.406561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.406761 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:08.406871 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:08.407395 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 10:35:08.407513 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:35:08.408505 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:08.395846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39261 (local address 127.2.193.131:35531)
0924 10:35:08.396301 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:08.396350 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:08.397666 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:08.398238 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:08.398245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:08.398598 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:08.399082 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:08.399095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.399896 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.399902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:08.400783 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:08.400791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.401351 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.401355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:08.401548 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:08.402228 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:08.402244 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:08.404016 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:08.405732 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:08.405738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:08.405754 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:08.406021 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:08.406313 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:08.406316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:08.406318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:08.406459 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:08.406871 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:08.406880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:08.407132 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 10:35:08.407338 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 10:35:08.407675 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260924 10:35:08.409660 6885 heartbeater.cc:347] Connected to a master server at 127.2.193.190:39261
I20260924 10:35:08.409916 6885 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:08.410411 6885 heartbeater.cc:511] Master 127.2.193.190:39261 requested a full tablet report, sending...
I20260924 10:35:08.411430 6426 ts_manager.cc:194] Registered new tserver with Master: 119456d5c8a3480783ef5cfd85d7eced (127.2.193.131:45921)
I20260924 10:35:08.412012 6426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:35531
I20260924 10:35:08.416065 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:35:08.441713 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:08.431981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.2.193.129:45563)
0924 10:35:08.432304 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:08.432323 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:08.432347 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:35:08.432399 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:08.432404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:08.432607 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:08.432785 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:08.433545 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.434372 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.435314 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.435708 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.435830 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:08.438282 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:08.438290 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:08.438304 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:08.438328 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:08.439781 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.440273 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.440279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.440280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.440359 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.440642 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.440647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.440648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.440895 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:08.441142 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:08.441374 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:35:08.441542 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":130,"threads_started":1}
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:08.452404 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:08.444484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.2.193.190:39261)
0924 10:35:08.444618 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:08.444623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:08.444688 (+ 65us) server_negotiation.cc:407] Connection header received
0924 10:35:08.444825 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:08.444828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:08.444877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:08.444962 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:08.445612 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.446122 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.446900 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:08.447126 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:08.447236 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:08.449568 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:08.449572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:08.449574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:08.449594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:08.450793 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.451197 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.451201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.451202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.451279 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:08.451544 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:08.451548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:08.451550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:08.451761 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:08.451902 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:08.452123 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 10:35:08.452237 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 10:35:08.452817 6426 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932
I20260924 10:35:08.453462 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 6488
I20260924 10:35:08.459120 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 6622
I20260924 10:35:08.464007 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 6756
I20260924 10:35:08.469264 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 6394
2026-09-24T10:35:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2156 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6897](info): set up 2 sockets
Sep 24 10:35:08 dist-test-slave-8wc7 krb5kdc[6897](info): commencing operation
krb5kdc: starting...
W20260924 10:35:09.505467 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:09 dist-test-slave-8wc7 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:09.629304 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:37571
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:37571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:09.743053 6910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:09.743350 6910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:09.743394 6910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:09.747778 6910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:09.747850 6910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:09.747869 6910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:09.747891 6910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:09.747908 6910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:09.753968 6910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:37571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:37571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:09.755252 6910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:09.756378 6910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:09.765271 6918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:09.765275 6915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:09.765339 6916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:09.765424 6910 server_base.cc:1034] running on GCE node
I20260924 10:35:09.766280 6910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:09.767552 6910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:09.768716 6910 hybrid_clock.cc:648] HybridClock initialized: now 1790246109768683 us; error 51 us; skew 500 ppm
Sep 24 10:35:09 dist-test-slave-8wc7 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246109, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:09.776146 6910 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:09.777055 6910 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:09.777129 6910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:09.777431 6910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:09.779610 6910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "e59bcd89c5d648f8ba5345088b8c55dd"
format_stamp: "Formatted at 2026-09-24 10:35:09 on dist-test-slave-8wc7"
server_key: "c0ec3db2be5e8843efb10889e00f898d"
server_key_iv: "b53e5679e6a51e881ed134f1a4fa596a"
server_key_version: "encryptionkey@0"
I20260924 10:35:09.780220 6910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "e59bcd89c5d648f8ba5345088b8c55dd"
format_stamp: "Formatted at 2026-09-24 10:35:09 on dist-test-slave-8wc7"
server_key: "c0ec3db2be5e8843efb10889e00f898d"
server_key_iv: "b53e5679e6a51e881ed134f1a4fa596a"
server_key_version: "encryptionkey@0"
I20260924 10:35:09.784176 6910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 10:35:09.786876 6924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:09.788565 6910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:35:09.788720 6910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "e59bcd89c5d648f8ba5345088b8c55dd"
format_stamp: "Formatted at 2026-09-24 10:35:09 on dist-test-slave-8wc7"
server_key: "c0ec3db2be5e8843efb10889e00f898d"
server_key_iv: "b53e5679e6a51e881ed134f1a4fa596a"
server_key_version: "encryptionkey@0"
I20260924 10:35:09.788861 6910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:09.802865 6910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:09.806018 6910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:09.806239 6910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:09.813580 6910 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:37571
I20260924 10:35:09.813848 6976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:37571 every 8 connection(s)
I20260924 10:35:09.814328 6910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:09.816169 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 6910
I20260924 10:35:09.816303 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:09.816615 2822 external_mini_cluster.cc:1428] Setting key eac617989474a269c59b22a3ca25a3a7
I20260924 10:35:09.817610 6977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:09.823542 6977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd: Bootstrap starting.
Sep 24 10:35:09 dist-test-slave-8wc7 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:09.826364 6977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:09.827303 6977 log.cc:824] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:09.829941 6977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd: No bootstrap required, opened a new log
I20260924 10:35:09.832194 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:09.818087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.2.193.190:37571)
0924 10:35:09.818538 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:09.818552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:09.818589 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:35:09.819251 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:09.819268 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:09.819612 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:09.820014 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:09.820858 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:09.821916 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:09.822810 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:09.823308 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:09.823485 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:09.826531 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:09.826540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:09.826567 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:09.826601 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:09.828752 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:09.829324 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:09.829330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:09.829337 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:09.829483 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:09.829776 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:09.829779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:09.829781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:09.830333 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:09.830531 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:09.830834 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 10:35:09.831161 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":152,"threads_started":1}
I20260924 10:35:09.834441 6977 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37571 } }
I20260924 10:35:09.834782 6977 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:09.834852 6977 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59bcd89c5d648f8ba5345088b8c55dd, State: Initialized, Role: FOLLOWER
I20260924 10:35:09.835249 6977 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37571 } }
I20260924 10:35:09.835376 6977 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:09.835423 6977 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:09.835494 6977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:09.836522 6977 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37571 } }
I20260924 10:35:09.836865 6977 leader_election.cc:302] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59bcd89c5d648f8ba5345088b8c55dd; no voters:
I20260924 10:35:09.837155 6977 leader_election.cc:288] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:09.837378 6982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:09.837610 6982 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [term 1 LEADER]: Becoming Leader. State: Replica: e59bcd89c5d648f8ba5345088b8c55dd, State: Running, Role: LEADER
I20260924 10:35:09.837918 6982 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37571 } }
I20260924 10:35:09.838538 6977 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:09.839839 6984 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59bcd89c5d648f8ba5345088b8c55dd. Latest consensus state: current_term: 1 leader_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37571 } } }
I20260924 10:35:09.839975 6984 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:09.840325 6991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:09.840955 6983 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59bcd89c5d648f8ba5345088b8c55dd" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37571 } } }
I20260924 10:35:09.841060 6983 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:09.843750 6991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:09.850615 6991 catalog_manager.cc:1347] Generated new cluster ID: ff0e801107bf4db0b4659177b4e20e15
I20260924 10:35:09.850713 6991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:09.871395 6991 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:09.872774 6991 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:09.878640 6991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd: Generated new TSK 0
I20260924 10:35:09.879438 6991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:35:09.922610 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37571
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:10.067193 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:10.067552 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:10.067601 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:10.071826 7003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:10.071902 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:10.071967 7003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:10.077513 7003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:10.078629 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:10.079958 7003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:10.089531 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.089561 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.089531 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:10.089815 7003 server_base.cc:1034] running on GCE node
I20260924 10:35:10.090411 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:10.091152 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:10.092375 7003 hybrid_clock.cc:648] HybridClock initialized: now 1790246110092344 us; error 32 us; skew 500 ppm
Sep 24 10:35:10 dist-test-slave-8wc7 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246110, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:10.095400 7003 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:35:10.096220 7003 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:10.096315 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:10.096560 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:10.098735 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "d985733de42b4efeb917682bc1a0303e"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "50cf722a4f8db420a6d60f1e694c3fa2"
server_key_iv: "c8dda2ed9a2aec75309dda80a67efb28"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.099337 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "d985733de42b4efeb917682bc1a0303e"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "50cf722a4f8db420a6d60f1e694c3fa2"
server_key_iv: "c8dda2ed9a2aec75309dda80a67efb28"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.103327 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:10.105846 7017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.107117 7003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:10.107281 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "d985733de42b4efeb917682bc1a0303e"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "50cf722a4f8db420a6d60f1e694c3fa2"
server_key_iv: "c8dda2ed9a2aec75309dda80a67efb28"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.107406 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:10.131511 7003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:10.134843 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:10.135069 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:10.135710 7003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:10.136579 7003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:10.136653 7003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.136731 7003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:10.136775 7003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.146044 7003 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38999
I20260924 10:35:10.146059 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38999 every 8 connection(s)
I20260924 10:35:10.146692 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:10.150748 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7003
I20260924 10:35:10.150858 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:10.151157 2822 external_mini_cluster.cc:1428] Setting key 7ae5580065a79e0a8cfc253443661588
Sep 24 10:35:10 dist-test-slave-8wc7 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246110, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:35:10.161769 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.148625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:57833 (local address 127.2.193.190:37571)
0924 10:35:10.148777 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.148781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.149613 (+ 832us) server_negotiation.cc:407] Connection header received
0924 10:35:10.151022 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.151025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.151084 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.151196 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:10.152654 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.153256 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.154452 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.154692 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.154806 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.157755 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:10.157758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:10.157761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:10.157785 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:10.159345 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.159966 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.159972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.159976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.160061 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.160421 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.160424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.160426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.160672 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:10.160778 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.161411 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.161564 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:35:10.162683 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.148852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37571 (local address 127.2.193.129:57833)
0924 10:35:10.149441 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.149488 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.150747 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.151400 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.151409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.151871 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:10.152510 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.152528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.153386 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.153390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.154309 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.154318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.154851 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.154855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.155039 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.155674 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:10.155699 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:10.157554 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:10.159542 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:10.159549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:10.159567 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:10.159852 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:10.160155 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:10.160160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:10.160162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:10.160317 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:10.160790 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:10.160799 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:10.161081 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.161373 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.161738 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:10.164016 7131 heartbeater.cc:347] Connected to a master server at 127.2.193.190:37571
I20260924 10:35:10.164297 7131 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:10.164785 7131 heartbeater.cc:511] Master 127.2.193.190:37571 requested a full tablet report, sending...
I20260924 10:35:10.166435 6941 ts_manager.cc:194] Registered new tserver with Master: d985733de42b4efeb917682bc1a0303e (127.2.193.129:38999)
I20260924 10:35:10.167650 6941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:57833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:35:10.180859 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37571
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:10.298177 7136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:10.298480 7136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:10.298527 7136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:10.302711 7136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:10.302783 7136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:10.302846 7136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:10.308468 7136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:10.309633 7136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:10.310714 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:10.320663 7141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.320645 7144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.320645 7142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:10.321043 7136 server_base.cc:1034] running on GCE node
I20260924 10:35:10.321535 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:10.322237 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:10.323431 7136 hybrid_clock.cc:648] HybridClock initialized: now 1790246110323410 us; error 35 us; skew 500 ppm
Sep 24 10:35:10 dist-test-slave-8wc7 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246110, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:10.326375 7136 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:35:10.327172 7136 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:10.327301 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:10.327554 7136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:10.329804 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "e69c4961fe094ff08b778486d56671e3"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "e41bd7e0e1ac22d66277b6cda04f95cb"
server_key_iv: "bff557673d3098dce55f5f2be5f977a4"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.330436 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "e69c4961fe094ff08b778486d56671e3"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "e41bd7e0e1ac22d66277b6cda04f95cb"
server_key_iv: "bff557673d3098dce55f5f2be5f977a4"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.334367 7136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:10.336938 7150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.338482 7136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:10.338619 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "e69c4961fe094ff08b778486d56671e3"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "e41bd7e0e1ac22d66277b6cda04f95cb"
server_key_iv: "bff557673d3098dce55f5f2be5f977a4"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.338788 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:10.358173 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:10.361238 7136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:10.361480 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:10.362100 7136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:10.362968 7136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:10.363041 7136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.363111 7136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:10.363159 7136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.371821 7136 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:44679
I20260924 10:35:10.371837 7263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:44679 every 8 connection(s)
I20260924 10:35:10.372443 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:10.378487 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7136
I20260924 10:35:10.378578 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:10.378882 2822 external_mini_cluster.cc:1428] Setting key ce31fdcacb8608fc485d9ce78a65bfe1
Sep 24 10:35:10 dist-test-slave-8wc7 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246110, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:10.387338 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.374218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:49671 (local address 127.2.193.190:37571)
0924 10:35:10.374361 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.374365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.375209 (+ 844us) server_negotiation.cc:407] Connection header received
0924 10:35:10.376610 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.376613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.376675 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.376772 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:10.378040 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.378610 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.380011 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.380346 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.380474 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.383268 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:10.383275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:10.383282 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:10.383316 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:10.385027 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.385610 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.385617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.385621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.385722 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.386083 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.386088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.386092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.386366 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:10.386486 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.386984 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.387133 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:35:10.388162 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.374621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37571 (local address 127.2.193.130:49671)
0924 10:35:10.375059 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.375103 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.376381 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.376950 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.376957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.377394 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:10.377889 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.377902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.378742 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.378748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.379888 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.379897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.380514 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.380518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.380690 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.381328 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:10.381354 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:10.383087 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:10.385144 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:10.385149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:10.385162 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:10.385496 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:10.385827 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:10.385829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:10.385830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:10.385975 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:10.386476 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:10.386482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:10.386726 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.386878 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.387157 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:10.389542 7264 heartbeater.cc:347] Connected to a master server at 127.2.193.190:37571
I20260924 10:35:10.389794 7264 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:10.390268 7264 heartbeater.cc:511] Master 127.2.193.190:37571 requested a full tablet report, sending...
I20260924 10:35:10.391340 6941 ts_manager.cc:194] Registered new tserver with Master: e69c4961fe094ff08b778486d56671e3 (127.2.193.130:44679)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
I20260924 10:35:10.392136 6941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:49671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:35:10.407966 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37571
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:10.534276 7269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:10.534723 7269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:10.534816 7269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:10.540603 7269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:10.540699 7269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:10.540791 7269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:10.546617 7269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:10.547837 7269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:10.548895 7269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:10.558897 7269 server_base.cc:1034] running on GCE node
W20260924 10:35:10.559072 7277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.558847 7274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.558847 7275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:10.559557 7269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:10.560225 7269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:10.561414 7269 hybrid_clock.cc:648] HybridClock initialized: now 1790246110561378 us; error 52 us; skew 500 ppm
Sep 24 10:35:10 dist-test-slave-8wc7 krb5kdc[6897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246110, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:10.564195 7269 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:35:10.564985 7269 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:10.565083 7269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:10.565353 7269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:10.567431 7269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "80646fb791324a3d8faa10a148fb3a44"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "881e565574c5f3df37889b79d356c9a3"
server_key_iv: "43dd8988dd70cb217b7dc13ff27f4758"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.568018 7269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "80646fb791324a3d8faa10a148fb3a44"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "881e565574c5f3df37889b79d356c9a3"
server_key_iv: "43dd8988dd70cb217b7dc13ff27f4758"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.571887 7269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:10.574479 7283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.575727 7269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:10.575892 7269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "80646fb791324a3d8faa10a148fb3a44"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "881e565574c5f3df37889b79d356c9a3"
server_key_iv: "43dd8988dd70cb217b7dc13ff27f4758"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.576015 7269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:10.614297 7269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:10.617416 7269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:10.617626 7269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:10.618224 7269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:10.619088 7269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:10.619161 7269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.619231 7269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:10.619266 7269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.628692 7269 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:33837
I20260924 10:35:10.628705 7396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:33837 every 8 connection(s)
I20260924 10:35:10.629364 7269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:10.635077 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7269
I20260924 10:35:10.635208 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:10.635497 2822 external_mini_cluster.cc:1428] Setting key a2347c7f5eefd9f51da2b153f97ce389
Sep 24 10:35:10 dist-test-slave-8wc7 krb5kdc[6897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246110, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:10.643770 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.631230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:46037 (local address 127.2.193.190:37571)
0924 10:35:10.631371 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.631382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.632134 (+ 752us) server_negotiation.cc:407] Connection header received
0924 10:35:10.633584 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.633587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.633649 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.633759 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:10.634995 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.635683 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.637161 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.637475 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.637577 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.640261 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:10.640264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:10.640266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:10.640289 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:10.641813 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.642348 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.642352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.642354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.642439 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.642754 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.642756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.642758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.642935 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:10.643020 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.643539 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.643647 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:35:10.644666 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.631533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37571 (local address 127.2.193.131:46037)
0924 10:35:10.631975 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.632018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.633342 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.633948 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.633955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.634321 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:10.634825 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.634838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.635818 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.635822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.637031 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.637041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.637635 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.637641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.637811 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.638457 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:10.638495 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:10.640099 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:10.641922 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:10.641927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:10.641939 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:10.642196 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:10.642526 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:10.642529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:10.642530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:10.642672 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:10.643042 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:10.643047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:10.643287 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.643489 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.643779 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260924 10:35:10.645895 7397 heartbeater.cc:347] Connected to a master server at 127.2.193.190:37571
I20260924 10:35:10.646153 7397 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:10.646636 7397 heartbeater.cc:511] Master 127.2.193.190:37571 requested a full tablet report, sending...
I20260924 10:35:10.647792 6941 ts_manager.cc:194] Registered new tserver with Master: 80646fb791324a3d8faa10a148fb3a44 (127.2.193.131:33837)
I20260924 10:35:10.648480 6941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:46037
I20260924 10:35:10.649574 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:35:10.658571 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.651508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.2.193.190:37571)
0924 10:35:10.651635 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.651639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.651858 (+ 219us) server_negotiation.cc:407] Connection header received
0924 10:35:10.651994 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.651997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.652039 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.652114 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:10.652776 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.653375 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.654158 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.654375 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.654457 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.655723 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:10.655726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:10.655728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:10.655749 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:10.657011 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.657501 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.657504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.657506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.657584 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:10.657837 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:10.657839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:10.657840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:10.658047 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:10.658154 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.658350 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.658451 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:35:10.661652 6941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:10.664724 6941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:10.679968 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.674550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38999 (local address 127.0.0.1:43126)
0924 10:35:10.674806 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.674858 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.674999 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.675679 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.675689 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.675714 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:10.675857 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.675868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.676973 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.676980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.678375 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.678386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.679377 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.679380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.679501 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.679544 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.679663 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.679789 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":14,"spinlock_wait_cycles":3200,"thread_start_us":75,"threads_started":1}
I20260924 10:35:10.680202 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.674621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.2.193.130:44679)
0924 10:35:10.675348 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.675358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.675377 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:35:10.675571 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.675579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.675743 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.675882 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:10.676162 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.677407 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.678838 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.679633 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.679758 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.679894 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.679958 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.680062 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":112,"threads_started":1}
I20260924 10:35:10.680480 7409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.673782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:33837 (local address 127.0.0.1:57936)
0924 10:35:10.675140 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.675150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.675232 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.675928 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.675934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.675955 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:10.676105 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.676111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.677548 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.677551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.678675 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.678682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.680037 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.680042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.680159 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.680179 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.680284 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.680348 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":75,"thread_start_us":94,"threads_started":1}
I20260924 10:35:10.680621 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.673917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.2.193.131:33837)
0924 10:35:10.674311 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.674325 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.675223 (+ 898us) server_negotiation.cc:407] Connection header received
0924 10:35:10.675390 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.675397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.675605 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.675738 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:10.676246 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.677407 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.678796 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.679650 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.679758 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.679894 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.680360 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.680468 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":194,"threads_started":1}
I20260924 10:35:10.680200 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.673823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:44679 (local address 127.0.0.1:44300)
0924 10:35:10.674803 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.674858 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.674999 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.675867 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.675870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.675889 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:10.676051 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.676058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.677581 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.677584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.678712 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.678723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.679752 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.679755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.679832 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.679851 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.679910 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.679974 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":28,"spinlock_wait_cycles":6272,"thread_start_us":63,"threads_started":1}
I20260924 10:35:10.680981 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.674727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.2.193.129:38999)
0924 10:35:10.675089 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.675096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.675118 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:35:10.675269 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.675275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.675510 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.675659 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:10.675995 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.676849 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.678487 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.679223 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.679357 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.679520 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.680612 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.680838 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":80,"threads_started":1}
I20260924 10:35:10.683533 7331 tablet_service.cc:1467] Processing CreateTablet for tablet c087a5294bb74d66988386aa1fbd9bc2 (DEFAULT_TABLE table=test-table [id=77ebc9a57efa486280698a53c42d33ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:10.683638 7065 tablet_service.cc:1467] Processing CreateTablet for tablet c087a5294bb74d66988386aa1fbd9bc2 (DEFAULT_TABLE table=test-table [id=77ebc9a57efa486280698a53c42d33ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:10.683638 7198 tablet_service.cc:1467] Processing CreateTablet for tablet c087a5294bb74d66988386aa1fbd9bc2 (DEFAULT_TABLE table=test-table [id=77ebc9a57efa486280698a53c42d33ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:10.684584 7331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c087a5294bb74d66988386aa1fbd9bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:10.684628 7065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c087a5294bb74d66988386aa1fbd9bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:10.684628 7198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c087a5294bb74d66988386aa1fbd9bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:10.691370 7414 tablet_bootstrap.cc:492] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Bootstrap starting.
I20260924 10:35:10.692371 7415 tablet_bootstrap.cc:492] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e: Bootstrap starting.
I20260924 10:35:10.692647 7416 tablet_bootstrap.cc:492] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Bootstrap starting.
I20260924 10:35:10.693859 7414 tablet_bootstrap.cc:654] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:10.694602 7415 tablet_bootstrap.cc:654] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:10.694602 7416 tablet_bootstrap.cc:654] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:10.695044 7414 log.cc:824] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:10.695477 7415 log.cc:824] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:10.695814 7416 log.cc:824] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:10.697283 7414 tablet_bootstrap.cc:492] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: No bootstrap required, opened a new log
I20260924 10:35:10.697446 7415 tablet_bootstrap.cc:492] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e: No bootstrap required, opened a new log
I20260924 10:35:10.697497 7416 tablet_bootstrap.cc:492] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: No bootstrap required, opened a new log
I20260924 10:35:10.697513 7414 ts_tablet_manager.cc:1397] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:35:10.697630 7416 ts_tablet_manager.cc:1397] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:35:10.697630 7415 ts_tablet_manager.cc:1397] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260924 10:35:10.700397 7414 raft_consensus.cc:348] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.700677 7414 raft_consensus.cc:374] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:10.700738 7414 raft_consensus.cc:729] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80646fb791324a3d8faa10a148fb3a44, State: Initialized, Role: FOLLOWER
I20260924 10:35:10.701224 7414 consensus_queue.cc:260] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.701547 7416 raft_consensus.cc:348] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.701817 7416 raft_consensus.cc:374] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:10.701872 7416 raft_consensus.cc:729] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69c4961fe094ff08b778486d56671e3, State: Initialized, Role: FOLLOWER
I20260924 10:35:10.702001 7397 heartbeater.cc:503] Master 127.2.193.190:37571 was elected leader, sending a full tablet report...
I20260924 10:35:10.702337 7414 ts_tablet_manager.cc:1428] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:35:10.702347 7416 consensus_queue.cc:260] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.702596 7415 raft_consensus.cc:348] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.702852 7415 raft_consensus.cc:374] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:10.702909 7415 raft_consensus.cc:729] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d985733de42b4efeb917682bc1a0303e, State: Initialized, Role: FOLLOWER
I20260924 10:35:10.703306 7416 ts_tablet_manager.cc:1428] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:35:10.703331 7415 consensus_queue.cc:260] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.704034 7264 heartbeater.cc:503] Master 127.2.193.190:37571 was elected leader, sending a full tablet report...
I20260924 10:35:10.704242 7415 ts_tablet_manager.cc:1428] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:35:10.704372 7131 heartbeater.cc:503] Master 127.2.193.190:37571 was elected leader, sending a full tablet report...
I20260924 10:35:10.710669 7422 raft_consensus.cc:482] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:10.710855 7422 raft_consensus.cc:504] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.712193 7422 leader_election.cc:288] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80646fb791324a3d8faa10a148fb3a44 (127.2.193.131:33837), e69c4961fe094ff08b778486d56671e3 (127.2.193.130:44679)
I20260924 10:35:10.716176 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.712574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:33769 (local address 127.2.193.130:44679)
0924 10:35:10.712694 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.712699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.713005 (+ 306us) server_negotiation.cc:407] Connection header received
0924 10:35:10.713108 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.713111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.713158 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.713247 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:10.713529 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.713995 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.715128 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.715543 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.715674 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.715728 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.715978 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.716047 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:35:10.716179 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.712469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:57235 (local address 127.2.193.131:33837)
0924 10:35:10.712614 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.712620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.712852 (+ 232us) server_negotiation.cc:407] Connection header received
0924 10:35:10.712947 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.712951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.713002 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.713076 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:10.713417 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.713968 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.715129 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.715546 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.715671 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.715728 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.715997 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.716065 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:35:10.716204 7424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.712361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:33837 (local address 127.2.193.129:57235)
0924 10:35:10.712748 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.712772 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.712863 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.713098 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.713100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.713117 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:10.713293 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.713301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.714088 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.714094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.715002 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.715011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.715674 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.715678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.715840 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.715869 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.715976 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.716047 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":75,"thread_start_us":131,"threads_started":1}
I20260924 10:35:10.716209 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.712470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:44679 (local address 127.2.193.129:33769)
0924 10:35:10.712884 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.712899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:10.713002 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:10.713270 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:10.713273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:10.713289 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:10.713416 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.713423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.714143 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.714146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.714999 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:10.715008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.715667 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.715670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:10.715847 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.715870 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:35:10.715963 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:35:10.716039 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":59,"thread_start_us":122,"threads_started":1}
I20260924 10:35:10.717054 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c087a5294bb74d66988386aa1fbd9bc2" candidate_uuid: "d985733de42b4efeb917682bc1a0303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80646fb791324a3d8faa10a148fb3a44" is_pre_election: true
I20260924 10:35:10.717387 7351 raft_consensus.cc:2454] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d985733de42b4efeb917682bc1a0303e in term 0.
I20260924 10:35:10.717422 7218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c087a5294bb74d66988386aa1fbd9bc2" candidate_uuid: "d985733de42b4efeb917682bc1a0303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69c4961fe094ff08b778486d56671e3" is_pre_election: true
I20260924 10:35:10.717634 7218 raft_consensus.cc:2454] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d985733de42b4efeb917682bc1a0303e in term 0.
I20260924 10:35:10.717854 7021 leader_election.cc:302] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80646fb791324a3d8faa10a148fb3a44, d985733de42b4efeb917682bc1a0303e; no voters:
I20260924 10:35:10.718142 7422 raft_consensus.cc:2781] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:10.718240 7422 raft_consensus.cc:482] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:10.718300 7422 raft_consensus.cc:3037] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:10.719316 7422 raft_consensus.cc:504] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.719678 7422 leader_election.cc:288] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [CANDIDATE]: Term 1 election: Requested vote from peers 80646fb791324a3d8faa10a148fb3a44 (127.2.193.131:33837), e69c4961fe094ff08b778486d56671e3 (127.2.193.130:44679)
I20260924 10:35:10.720057 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c087a5294bb74d66988386aa1fbd9bc2" candidate_uuid: "d985733de42b4efeb917682bc1a0303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80646fb791324a3d8faa10a148fb3a44"
I20260924 10:35:10.720114 7218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c087a5294bb74d66988386aa1fbd9bc2" candidate_uuid: "d985733de42b4efeb917682bc1a0303e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e69c4961fe094ff08b778486d56671e3"
I20260924 10:35:10.720202 7351 raft_consensus.cc:3037] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:10.720244 7218 raft_consensus.cc:3037] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:10.721346 7218 raft_consensus.cc:2454] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d985733de42b4efeb917682bc1a0303e in term 1.
I20260924 10:35:10.721350 7351 raft_consensus.cc:2454] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d985733de42b4efeb917682bc1a0303e in term 1.
I20260924 10:35:10.721665 7021 leader_election.cc:302] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80646fb791324a3d8faa10a148fb3a44, d985733de42b4efeb917682bc1a0303e; no voters:
I20260924 10:35:10.721896 7422 raft_consensus.cc:2781] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:10.722203 7422 raft_consensus.cc:686] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 1 LEADER]: Becoming Leader. State: Replica: d985733de42b4efeb917682bc1a0303e, State: Running, Role: LEADER
I20260924 10:35:10.722564 7422 consensus_queue.cc:237] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } }
I20260924 10:35:10.726024 6941 catalog_manager.cc:5495] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e reported cstate change: term changed from 0 to 1, leader changed from <none> to d985733de42b4efeb917682bc1a0303e (127.2.193.129). New cstate: current_term: 1 leader_uuid: "d985733de42b4efeb917682bc1a0303e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d985733de42b4efeb917682bc1a0303e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38999 } health_report { overall_health: HEALTHY } } }
I20260924 10:35:10.744773 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.740648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.2.193.129:38999)
0924 10:35:10.740910 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.740913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.740926 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:35:10.741004 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.741007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.741059 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.741151 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:10.741492 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.741997 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.743301 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:10.743554 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:10.743694 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:10.743986 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:10.744414 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:10.744509 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.744610 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.744662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260924 10:35:10.750808 7218 raft_consensus.cc:1261] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 1 FOLLOWER]: Refusing update from remote peer d985733de42b4efeb917682bc1a0303e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:10.750808 7351 raft_consensus.cc:1261] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 1 FOLLOWER]: Refusing update from remote peer d985733de42b4efeb917682bc1a0303e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:10.751570 7422 consensus_queue.cc:1035] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [LEADER]: Connected to new peer: Peer: permanent_uuid: "80646fb791324a3d8faa10a148fb3a44" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:10.751827 7425 consensus_queue.cc:1035] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e69c4961fe094ff08b778486d56671e3" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 44679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:10.761055 7429 mvcc.cc:205] Tried to move back new op lower bound from 7332848069627760640 to 7332848069522726912. Current Snapshot: MvccSnapshot[applied={T|T < 7332848069627760640}]
I20260924 10:35:10.761292 7430 mvcc.cc:205] Tried to move back new op lower bound from 7332848069627760640 to 7332848069522726912. Current Snapshot: MvccSnapshot[applied={T|T < 7332848069627760640}]
I20260924 10:35:10.769531 6941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:10.769698 6941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:10.772181 6941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e59bcd89c5d648f8ba5345088b8c55dd: Sending DeleteTablet for 3 replicas of tablet c087a5294bb74d66988386aa1fbd9bc2
I20260924 10:35:10.773029 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet c087a5294bb74d66988386aa1fbd9bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:10 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:57936
I20260924 10:35:10.773077 7065 tablet_service.cc:1514] Processing DeleteTablet for tablet c087a5294bb74d66988386aa1fbd9bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:10 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:43126
I20260924 10:35:10.773403 7198 tablet_service.cc:1514] Processing DeleteTablet for tablet c087a5294bb74d66988386aa1fbd9bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:10 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:44300
I20260924 10:35:10.773622 7438 tablet_replica.cc:330] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e: stopping tablet replica
I20260924 10:35:10.773921 7439 tablet_replica.cc:330] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: stopping tablet replica
I20260924 10:35:10.773973 7438 raft_consensus.cc:2229] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:10.774315 7439 raft_consensus.cc:2229] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:10.774427 7438 raft_consensus.cc:2258] T c087a5294bb74d66988386aa1fbd9bc2 P d985733de42b4efeb917682bc1a0303e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:10.774746 7439 raft_consensus.cc:2258] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:10.774923 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7003
I20260924 10:35:10.775861 7440 tablet_replica.cc:330] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: stopping tablet replica
I20260924 10:35:10.776149 7440 raft_consensus.cc:2229] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:10.776477 7440 raft_consensus.cc:2258] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:10.778043 7439 ts_tablet_manager.cc:1905] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:10.778304 7440 ts_tablet_manager.cc:1905] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:10.780825 7440 ts_tablet_manager.cc:1918] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:10.780944 7440 log.cc:1196] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/c087a5294bb74d66988386aa1fbd9bc2
I20260924 10:35:10.781314 7440 ts_tablet_manager.cc:1939] T c087a5294bb74d66988386aa1fbd9bc2 P 80646fb791324a3d8faa10a148fb3a44: Deleting consensus metadata
I20260924 10:35:10.781517 7439 ts_tablet_manager.cc:1918] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:10.781626 7439 log.cc:1196] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/c087a5294bb74d66988386aa1fbd9bc2
I20260924 10:35:10.781965 7439 ts_tablet_manager.cc:1939] T c087a5294bb74d66988386aa1fbd9bc2 P e69c4961fe094ff08b778486d56671e3: Deleting consensus metadata
I20260924 10:35:10.782848 6928 catalog_manager.cc:4841] TS 80646fb791324a3d8faa10a148fb3a44 (127.2.193.131:33837): tablet c087a5294bb74d66988386aa1fbd9bc2 (table test-table [id=77ebc9a57efa486280698a53c42d33ab]) successfully deleted
W20260924 10:35:10.783067 6925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:38999 (error 108)
I20260924 10:35:10.783470 6925 catalog_manager.cc:4841] TS e69c4961fe094ff08b778486d56671e3 (127.2.193.130:44679): tablet c087a5294bb74d66988386aa1fbd9bc2 (table test-table [id=77ebc9a57efa486280698a53c42d33ab]) successfully deleted
I20260924 10:35:10.783977 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7136
I20260924 10:35:10.784055 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.783749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38999 (local address 127.0.0.1:43156)
0924 10:35:10.783897 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:10.783958 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:35:10.785259 6925 catalog_manager.cc:4584] TS d985733de42b4efeb917682bc1a0303e (127.2.193.129:38999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c087a5294bb74d66988386aa1fbd9bc2: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38999: connect: Connection refused (error 111)
I20260924 10:35:10.789503 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7269
I20260924 10:35:10.795092 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 6910
2026-09-24T10:35:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2326 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T10:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:10Z Disabled control of system clock
I20260924 10:35:10.825783 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46749
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:35:10.938129 7445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:10.938426 7445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:10.938467 7445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:10.942818 7445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:10.942889 7445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:10.942904 7445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:10.942921 7445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:10.942937 7445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:10.947736 7445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:10.948833 7445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:10.950244 7445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:10.958907 7451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.958882 7453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:10.958886 7450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:10.959324 7445 server_base.cc:1034] running on GCE node
I20260924 10:35:10.959924 7445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:10.961078 7445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:10.962276 7445 hybrid_clock.cc:648] HybridClock initialized: now 1790246110962237 us; error 54 us; skew 500 ppm
I20260924 10:35:10.964202 7445 webserver.cc:460] Webserver started at http://127.2.193.190:42057/ using document root <none> and password file <none>
I20260924 10:35:10.964809 7445 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:10.964903 7445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:10.965143 7445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:10.967367 7445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "b29c9e2f06d94efdbe78be35a8f6606d"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "b38dfb32eafdc053b227e9c0228464c8"
server_key_iv: "ad6bf5a106fdce61d1e08d38bf12ad1f"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.967990 7445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "b29c9e2f06d94efdbe78be35a8f6606d"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "b38dfb32eafdc053b227e9c0228464c8"
server_key_iv: "ad6bf5a106fdce61d1e08d38bf12ad1f"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.971890 7445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 10:35:10.974368 7459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:10.975685 7445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:10.975836 7445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "b29c9e2f06d94efdbe78be35a8f6606d"
format_stamp: "Formatted at 2026-09-24 10:35:10 on dist-test-slave-8wc7"
server_key: "b38dfb32eafdc053b227e9c0228464c8"
server_key_iv: "ad6bf5a106fdce61d1e08d38bf12ad1f"
server_key_version: "encryptionkey@0"
I20260924 10:35:10.975961 7445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:10.980767 7445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:10.981532 7445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:10.981726 7445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:10.988950 7445 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:46749
I20260924 10:35:10.988948 7511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:46749 every 8 connection(s)
I20260924 10:35:10.990080 7445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:10.991752 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7445
I20260924 10:35:10.991848 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:10.992133 2822 external_mini_cluster.cc:1428] Setting key 99a7d118c0d7ea79980dc3ea08ae4ee2
I20260924 10:35:10.993650 7512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:10.998555 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:10.993628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.2.193.190:46749)
0924 10:35:10.994369 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:10.994388 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:10.994432 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:35:10.995315 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:10.995338 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:10.995808 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:10.996200 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:10.996947 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:10.996953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:10.996969 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:10.996979 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:10.997043 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:10.997068 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:10.997244 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:10.997322 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:35:10.997608 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":147,"threads_started":1}
I20260924 10:35:11.000515 7512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d: Bootstrap starting.
I20260924 10:35:11.002789 7512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:11.003557 7512 log.cc:824] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:11.005760 7512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d: No bootstrap required, opened a new log
I20260924 10:35:11.009644 7512 raft_consensus.cc:348] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46749 } }
I20260924 10:35:11.009909 7512 raft_consensus.cc:374] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:11.009997 7512 raft_consensus.cc:729] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29c9e2f06d94efdbe78be35a8f6606d, State: Initialized, Role: FOLLOWER
I20260924 10:35:11.010440 7512 consensus_queue.cc:260] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46749 } }
I20260924 10:35:11.010604 7512 raft_consensus.cc:388] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:11.010691 7512 raft_consensus.cc:482] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:11.010813 7512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:11.011787 7512 raft_consensus.cc:504] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46749 } }
I20260924 10:35:11.012197 7512 leader_election.cc:302] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b29c9e2f06d94efdbe78be35a8f6606d; no voters:
I20260924 10:35:11.012602 7512 leader_election.cc:288] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:11.012712 7517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:11.012975 7517 raft_consensus.cc:686] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [term 1 LEADER]: Becoming Leader. State: Replica: b29c9e2f06d94efdbe78be35a8f6606d, State: Running, Role: LEADER
I20260924 10:35:11.013310 7517 consensus_queue.cc:237] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46749 } }
I20260924 10:35:11.013938 7512 sys_catalog.cc:564] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:11.014999 7518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46749 } } }
I20260924 10:35:11.015137 7518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:11.015455 7519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b29c9e2f06d94efdbe78be35a8f6606d. Latest consensus state: current_term: 1 leader_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29c9e2f06d94efdbe78be35a8f6606d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46749 } } }
I20260924 10:35:11.015547 7519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:11.015560 7526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:11.018857 7526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:11.025125 7526 catalog_manager.cc:1347] Generated new cluster ID: 85b22a751ec04f66b5a63dec5a4c32ce
I20260924 10:35:11.025265 7526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:11.043931 7526 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:11.044281 7526 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:11.052000 7526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d: Generated new TSK 0
I20260924 10:35:11.052760 7526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:35:11.065984 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46749
--builtin_ntp_servers=127.2.193.148:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:35:11.186404 7536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:11.186695 7536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:11.186738 7536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:11.191051 7536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:11.191121 7536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:11.191183 7536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:11.196272 7536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:11.197487 7536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:11.198931 7536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:11.208928 7542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:11.208916 7541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:11.208971 7544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:11.209105 7536 server_base.cc:1034] running on GCE node
I20260924 10:35:11.209649 7536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:11.210395 7536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:11.211565 7536 hybrid_clock.cc:648] HybridClock initialized: now 1790246111211528 us; error 50 us; skew 500 ppm
I20260924 10:35:11.213568 7536 webserver.cc:460] Webserver started at http://127.2.193.129:44507/ using document root <none> and password file <none>
I20260924 10:35:11.214221 7536 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:11.214288 7536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:11.214512 7536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:11.216872 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "0869ed4978e44fff87952d3241d56840"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "6cab50da397d72d127ddc354a647ed9c"
server_key_iv: "13a30ba6ec8490cab058bc76e363e6d9"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.217533 7536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "0869ed4978e44fff87952d3241d56840"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "6cab50da397d72d127ddc354a647ed9c"
server_key_iv: "13a30ba6ec8490cab058bc76e363e6d9"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.221728 7536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:35:11.224409 7550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.225821 7536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:11.225975 7536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "0869ed4978e44fff87952d3241d56840"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "6cab50da397d72d127ddc354a647ed9c"
server_key_iv: "13a30ba6ec8490cab058bc76e363e6d9"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.226100 7536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:11.230847 7536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:11.231500 7536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:11.231698 7536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:11.232270 7536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:11.233306 7536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:11.233371 7536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.233421 7536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:11.233451 7536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.242175 7536 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:43865
I20260924 10:35:11.242264 7663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:43865 every 8 connection(s)
I20260924 10:35:11.243172 7536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:11.249526 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.245052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:43175 (local address 127.2.193.190:46749)
0924 10:35:11.245283 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.245288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.246344 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 10:35:11.247744 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.247748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.247793 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.247908 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.249004 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.249007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.249010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.249030 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.249050 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.249058 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.249162 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.249333 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.249383 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:35:11.250420 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.245499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46749 (local address 127.2.193.129:43175)
0924 10:35:11.246178 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.246221 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.247505 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.248083 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.248090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.248185 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.248737 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.248739 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.248780 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.248849 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.249142 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.249151 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.249157 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.249402 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.249657 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260924 10:35:11.251600 7664 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46749
I20260924 10:35:11.251843 7664 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:11.252264 7664 heartbeater.cc:511] Master 127.2.193.190:46749 requested a full tablet report, sending...
I20260924 10:35:11.253451 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7536
I20260924 10:35:11.253584 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:11.253852 2822 external_mini_cluster.cc:1428] Setting key 46817af0135758fb0df7e97e8c6dc7b6
I20260924 10:35:11.253952 7476 ts_manager.cc:194] Registered new tserver with Master: 0869ed4978e44fff87952d3241d56840 (127.2.193.129:43865)
I20260924 10:35:11.255973 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46749
--builtin_ntp_servers=127.2.193.148:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:35:11.370122 7667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:11.370415 7667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:11.370456 7667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:11.374531 7667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:11.374600 7667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:11.374660 7667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:11.379665 7667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:11.380900 7667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:11.382337 7667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:11.392215 7675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:11.392232 7667 server_base.cc:1034] running on GCE node
W20260924 10:35:11.392195 7673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:11.392195 7672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:11.392866 7667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:11.393601 7667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:11.394800 7667 hybrid_clock.cc:648] HybridClock initialized: now 1790246111394765 us; error 55 us; skew 500 ppm
I20260924 10:35:11.396684 7667 webserver.cc:460] Webserver started at http://127.2.193.130:36525/ using document root <none> and password file <none>
I20260924 10:35:11.397348 7667 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:11.397441 7667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:11.397651 7667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:11.399712 7667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "ad2d7819b135414d8ac89058eb71e7e2"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "260704a49d65019259c734fe230a1ce3"
server_key_iv: "869b9366ed1e8a91205967782b96b2bb"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.400276 7667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "ad2d7819b135414d8ac89058eb71e7e2"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "260704a49d65019259c734fe230a1ce3"
server_key_iv: "869b9366ed1e8a91205967782b96b2bb"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.404134 7667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 10:35:11.406723 7681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.407943 7667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:11.408069 7667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "ad2d7819b135414d8ac89058eb71e7e2"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "260704a49d65019259c734fe230a1ce3"
server_key_iv: "869b9366ed1e8a91205967782b96b2bb"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.408183 7667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:11.412851 7667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:11.413578 7667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:11.413789 7667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:11.414363 7667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:11.415354 7667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:11.415416 7667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.415479 7667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:11.415524 7667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.424045 7667 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:38685
I20260924 10:35:11.424065 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:38685 every 8 connection(s)
I20260924 10:35:11.425029 7667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:11.431255 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.426894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:40097 (local address 127.2.193.190:46749)
0924 10:35:11.427041 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.427046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.428180 (+ 1134us) server_negotiation.cc:407] Connection header received
0924 10:35:11.429503 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.429506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.429544 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.429623 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.430837 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.430840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.430842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.430852 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.430871 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.430896 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.430989 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.431110 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.431146 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:35:11.432168 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.427370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46749 (local address 127.2.193.130:40097)
0924 10:35:11.428017 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.428057 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.429301 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.429822 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.429829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.429926 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.430581 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.430582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.430623 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.430693 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.430991 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.430997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.431002 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.431130 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.431322 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260924 10:35:11.432358 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7667
I20260924 10:35:11.432476 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:11.432770 2822 external_mini_cluster.cc:1428] Setting key 0c2d2e8eb74f2bb873ed1ed4092036c9
I20260924 10:35:11.433454 7795 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46749
I20260924 10:35:11.433691 7795 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:11.434158 7795 heartbeater.cc:511] Master 127.2.193.190:46749 requested a full tablet report, sending...
I20260924 10:35:11.434815 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46749
--builtin_ntp_servers=127.2.193.148:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:35:11.435214 7475 ts_manager.cc:194] Registered new tserver with Master: ad2d7819b135414d8ac89058eb71e7e2 (127.2.193.130:38685)
W20260924 10:35:11.547344 7798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:11.547650 7798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:11.547693 7798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:11.551862 7798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:11.551929 7798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:11.551990 7798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:11.557282 7798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:11.558405 7798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:11.559811 7798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:11.569585 7806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:11.569603 7803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:11.569589 7804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:11.569841 7798 server_base.cc:1034] running on GCE node
I20260924 10:35:11.570405 7798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:11.571097 7798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:11.572316 7798 hybrid_clock.cc:648] HybridClock initialized: now 1790246111572297 us; error 29 us; skew 500 ppm
I20260924 10:35:11.574409 7798 webserver.cc:460] Webserver started at http://127.2.193.131:45077/ using document root <none> and password file <none>
I20260924 10:35:11.575053 7798 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:11.575151 7798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:11.575405 7798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:11.577912 7798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "eda75675fb3e499fb44ae0bed5a84fe1"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "21e26eb992988121b55f5eb72661b1f9"
server_key_iv: "7dccf1fa270a2e228ae743e41b3932cd"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.578516 7798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "eda75675fb3e499fb44ae0bed5a84fe1"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "21e26eb992988121b55f5eb72661b1f9"
server_key_iv: "7dccf1fa270a2e228ae743e41b3932cd"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.582388 7798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:11.584815 7812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.586133 7798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:11.586251 7798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "eda75675fb3e499fb44ae0bed5a84fe1"
format_stamp: "Formatted at 2026-09-24 10:35:11 on dist-test-slave-8wc7"
server_key: "21e26eb992988121b55f5eb72661b1f9"
server_key_iv: "7dccf1fa270a2e228ae743e41b3932cd"
server_key_version: "encryptionkey@0"
I20260924 10:35:11.586357 7798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:11.591355 7798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:11.591969 7798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:11.592139 7798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:11.592682 7798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:11.593652 7798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:11.593698 7798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.593762 7798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:11.593796 7798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:11.602232 7798 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:41769
I20260924 10:35:11.602252 7925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:41769 every 8 connection(s)
I20260924 10:35:11.603124 7798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:11.609057 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.604954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:48695 (local address 127.2.193.190:46749)
0924 10:35:11.605100 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.605105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.606293 (+ 1188us) server_negotiation.cc:407] Connection header received
0924 10:35:11.607556 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.607558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.607594 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.607672 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.608693 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.608695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.608697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.608707 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.608725 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.608732 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.608805 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.608912 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.608948 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:35:11.609946 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.605506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46749 (local address 127.2.193.131:48695)
0924 10:35:11.606151 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.606191 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.607360 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.607833 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.607839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.607926 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.608457 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.608458 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.608498 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.608563 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.608800 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.608805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.608809 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.608934 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.609149 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260924 10:35:11.611032 7926 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46749
I20260924 10:35:11.611232 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7798
I20260924 10:35:11.611269 7926 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:11.611358 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:11.611660 2822 external_mini_cluster.cc:1428] Setting key 0bc84493b8b2ab0b9f75749d0c4b9bd3
I20260924 10:35:11.611733 7926 heartbeater.cc:511] Master 127.2.193.190:46749 requested a full tablet report, sending...
I20260924 10:35:11.612751 7475 ts_manager.cc:194] Registered new tserver with Master: eda75675fb3e499fb44ae0bed5a84fe1 (127.2.193.131:41769)
I20260924 10:35:11.613615 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:35:11.617095 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.615730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.2.193.190:46749)
0924 10:35:11.615822 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.615825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.616050 (+ 225us) server_negotiation.cc:407] Connection header received
0924 10:35:11.616133 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.616134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.616163 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.616251 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.616846 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.616849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.616851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.616860 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.616876 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.616883 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.616954 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.616978 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.617007 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 10:35:11.619578 7476 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:11.622558 7476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:11.633612 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.630510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:38685 (local address 127.0.0.1:40594)
0924 10:35:11.631404 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.631417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.631550 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.632122 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.632128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.632206 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.632806 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.632808 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.632825 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.632879 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.633255 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.633264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.633269 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.633389 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.633460 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":86,"threads_started":1}
I20260924 10:35:11.633680 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.630666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.2.193.130:38685)
0924 10:35:11.631105 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.631118 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.631497 (+ 379us) server_negotiation.cc:407] Connection header received
0924 10:35:11.631651 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.631657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.631848 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.631985 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.633011 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.633017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.633029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.633041 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.633088 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.633126 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.633284 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.633414 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.633497 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":118,"threads_started":1}
I20260924 10:35:11.634238 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.630492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.2.193.129:43865)
0924 10:35:11.631305 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.631316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.631461 (+ 145us) server_negotiation.cc:407] Connection header received
0924 10:35:11.631590 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.631600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.631815 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.631959 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.633333 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.633342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.633355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.633377 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.633420 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.633453 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.633546 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.633981 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.634060 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":105,"threads_started":1}
I20260924 10:35:11.633895 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.630383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:43865 (local address 127.0.0.1:53168)
0924 10:35:11.631305 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.631358 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.631482 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.632701 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.632705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.632723 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.633161 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.633166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.633202 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.633234 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.633584 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.633586 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.633589 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.633660 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.633697 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":88,"threads_started":1}
I20260924 10:35:11.635597 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.631507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41769 (local address 127.0.0.1:42236)
0924 10:35:11.633040 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.633055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.633142 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.634331 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.634335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.634355 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.634772 (+ 417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.634774 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.634784 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.634808 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.635348 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.635351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.635354 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.635431 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.635473 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"thread_start_us":96,"threads_started":1}
I20260924 10:35:11.635665 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.631840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.2.193.131:41769)
0924 10:35:11.632426 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.632437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.633337 (+ 900us) server_negotiation.cc:407] Connection header received
0924 10:35:11.633410 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.633414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.633595 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.633728 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.635141 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.635147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.635158 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.635171 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.635217 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.635251 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.635374 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.635445 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.635511 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":157,"threads_started":1}
I20260924 10:35:11.637436 7598 tablet_service.cc:1467] Processing CreateTablet for tablet e0c975cc02fa4f42be6324b96acc3e52 (DEFAULT_TABLE table=test-table [id=8c467567136e4fc4924729cf70497f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:11.637483 7729 tablet_service.cc:1467] Processing CreateTablet for tablet e0c975cc02fa4f42be6324b96acc3e52 (DEFAULT_TABLE table=test-table [id=8c467567136e4fc4924729cf70497f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:11.638126 7860 tablet_service.cc:1467] Processing CreateTablet for tablet e0c975cc02fa4f42be6324b96acc3e52 (DEFAULT_TABLE table=test-table [id=8c467567136e4fc4924729cf70497f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:11.638594 7729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c975cc02fa4f42be6324b96acc3e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:11.638594 7598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c975cc02fa4f42be6324b96acc3e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:11.639015 7860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c975cc02fa4f42be6324b96acc3e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:11.646402 7945 tablet_bootstrap.cc:492] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Bootstrap starting.
I20260924 10:35:11.646592 7943 tablet_bootstrap.cc:492] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Bootstrap starting.
I20260924 10:35:11.646736 7944 tablet_bootstrap.cc:492] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: Bootstrap starting.
I20260924 10:35:11.648569 7943 tablet_bootstrap.cc:654] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:11.648731 7945 tablet_bootstrap.cc:654] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:11.648731 7944 tablet_bootstrap.cc:654] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:11.649464 7943 log.cc:824] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:11.649618 7945 log.cc:824] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:11.649619 7944 log.cc:824] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:11.651731 7943 tablet_bootstrap.cc:492] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: No bootstrap required, opened a new log
I20260924 10:35:11.651751 7944 tablet_bootstrap.cc:492] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: No bootstrap required, opened a new log
I20260924 10:35:11.651954 7943 ts_tablet_manager.cc:1397] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:35:11.651954 7944 ts_tablet_manager.cc:1397] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:35:11.652241 7945 tablet_bootstrap.cc:492] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: No bootstrap required, opened a new log
I20260924 10:35:11.652441 7945 ts_tablet_manager.cc:1397] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:35:11.656390 7945 raft_consensus.cc:348] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.656726 7945 raft_consensus.cc:374] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:11.656785 7945 raft_consensus.cc:729] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2d7819b135414d8ac89058eb71e7e2, State: Initialized, Role: FOLLOWER
I20260924 10:35:11.656631 7943 raft_consensus.cc:348] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.656735 7944 raft_consensus.cc:348] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.656925 7943 raft_consensus.cc:374] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:11.656939 7944 raft_consensus.cc:374] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:11.656981 7944 raft_consensus.cc:729] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0869ed4978e44fff87952d3241d56840, State: Initialized, Role: FOLLOWER
I20260924 10:35:11.656981 7943 raft_consensus.cc:729] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda75675fb3e499fb44ae0bed5a84fe1, State: Initialized, Role: FOLLOWER
I20260924 10:35:11.657322 7945 consensus_queue.cc:260] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.657467 7944 consensus_queue.cc:260] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.657467 7943 consensus_queue.cc:260] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.658427 7943 ts_tablet_manager.cc:1428] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:35:11.658427 7945 ts_tablet_manager.cc:1428] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:35:11.658520 7664 heartbeater.cc:503] Master 127.2.193.190:46749 was elected leader, sending a full tablet report...
I20260924 10:35:11.658749 7926 heartbeater.cc:503] Master 127.2.193.190:46749 was elected leader, sending a full tablet report...
I20260924 10:35:11.658761 7795 heartbeater.cc:503] Master 127.2.193.190:46749 was elected leader, sending a full tablet report...
I20260924 10:35:11.659678 7944 ts_tablet_manager.cc:1428] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
W20260924 10:35:11.676057 7796 tablet.cc:2369] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:35:11.744333 7665 tablet.cc:2369] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:35:11.822973 7950 raft_consensus.cc:482] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:11.823213 7950 raft_consensus.cc:504] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.824415 7950 leader_election.cc:288] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad2d7819b135414d8ac89058eb71e7e2 (127.2.193.130:38685), eda75675fb3e499fb44ae0bed5a84fe1 (127.2.193.131:41769)
I20260924 10:35:11.826596 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.824905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:54425 (local address 127.2.193.131:41769)
0924 10:35:11.825054 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.825061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.825101 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:35:11.825273 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.825276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.825315 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.825399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.826171 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.826174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.826176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.826186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.826202 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.826209 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.826289 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.826420 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.826459 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:35:11.826619 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.824694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:47055 (local address 127.2.193.130:38685)
0924 10:35:11.824832 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.824837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.825005 (+ 168us) server_negotiation.cc:407] Connection header received
0924 10:35:11.825089 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.825093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.825135 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.825259 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.825987 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.825991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.825994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.826007 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.826028 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.826039 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.826146 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.826450 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.826489 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:35:11.826658 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.824684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41769 (local address 127.2.193.129:54425)
0924 10:35:11.825027 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.825042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.825135 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.825417 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.825419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.825433 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.825996 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.825998 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.826008 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.826034 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.826316 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.826319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.826321 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.826412 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.826451 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":86,"threads_started":1}
I20260924 10:35:11.826846 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.824587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:38685 (local address 127.2.193.129:47055)
0924 10:35:11.824879 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.824895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:11.824999 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:11.825275 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:11.825278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:11.825293 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:11.825803 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:11.825806 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:11.825820 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:11.825855 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:11.826326 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:11.826329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:11.826332 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:35:11.826436 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:35:11.826472 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260924 10:35:11.827334 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c975cc02fa4f42be6324b96acc3e52" candidate_uuid: "0869ed4978e44fff87952d3241d56840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" is_pre_election: true
I20260924 10:35:11.827452 7749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c975cc02fa4f42be6324b96acc3e52" candidate_uuid: "0869ed4978e44fff87952d3241d56840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2d7819b135414d8ac89058eb71e7e2" is_pre_election: true
I20260924 10:35:11.827653 7880 raft_consensus.cc:2454] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0869ed4978e44fff87952d3241d56840 in term 0.
I20260924 10:35:11.827669 7749 raft_consensus.cc:2454] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0869ed4978e44fff87952d3241d56840 in term 0.
I20260924 10:35:11.828167 7554 leader_election.cc:302] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869ed4978e44fff87952d3241d56840, ad2d7819b135414d8ac89058eb71e7e2; no voters:
I20260924 10:35:11.828433 7950 raft_consensus.cc:2781] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:11.828523 7950 raft_consensus.cc:482] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:11.828580 7950 raft_consensus.cc:3037] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:11.829669 7950 raft_consensus.cc:504] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.830005 7950 leader_election.cc:288] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [CANDIDATE]: Term 1 election: Requested vote from peers ad2d7819b135414d8ac89058eb71e7e2 (127.2.193.130:38685), eda75675fb3e499fb44ae0bed5a84fe1 (127.2.193.131:41769)
I20260924 10:35:11.830341 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c975cc02fa4f42be6324b96acc3e52" candidate_uuid: "0869ed4978e44fff87952d3241d56840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda75675fb3e499fb44ae0bed5a84fe1"
I20260924 10:35:11.830380 7749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c975cc02fa4f42be6324b96acc3e52" candidate_uuid: "0869ed4978e44fff87952d3241d56840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad2d7819b135414d8ac89058eb71e7e2"
I20260924 10:35:11.830484 7749 raft_consensus.cc:3037] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:11.830475 7880 raft_consensus.cc:3037] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:11.831725 7749 raft_consensus.cc:2454] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0869ed4978e44fff87952d3241d56840 in term 1.
I20260924 10:35:11.831725 7880 raft_consensus.cc:2454] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0869ed4978e44fff87952d3241d56840 in term 1.
I20260924 10:35:11.832018 7554 leader_election.cc:302] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0869ed4978e44fff87952d3241d56840, ad2d7819b135414d8ac89058eb71e7e2; no voters:
I20260924 10:35:11.832207 7950 raft_consensus.cc:2781] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:11.832448 7950 raft_consensus.cc:686] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 1 LEADER]: Becoming Leader. State: Replica: 0869ed4978e44fff87952d3241d56840, State: Running, Role: LEADER
I20260924 10:35:11.832778 7950 consensus_queue.cc:237] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } }
I20260924 10:35:11.835786 7476 catalog_manager.cc:5495] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0869ed4978e44fff87952d3241d56840 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "0869ed4978e44fff87952d3241d56840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0869ed4978e44fff87952d3241d56840" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 } health_report { overall_health: UNKNOWN } } }
W20260924 10:35:11.854084 7927 tablet.cc:2369] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:35:11.874835 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.873229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.2.193.129:43865)
0924 10:35:11.873382 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:11.873388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:11.873402 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:11.873565 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:11.873568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:11.873627 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:11.873720 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:11.874433 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:11.874437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:11.874439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:11.874450 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:11.874475 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:11.874483 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:11.874573 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:11.874649 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 10:35:11.874689 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:35:11.880810 7749 raft_consensus.cc:1261] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 1 FOLLOWER]: Refusing update from remote peer 0869ed4978e44fff87952d3241d56840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:11.880810 7880 raft_consensus.cc:1261] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 1 FOLLOWER]: Refusing update from remote peer 0869ed4978e44fff87952d3241d56840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:11.881623 7954 consensus_queue.cc:1035] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eda75675fb3e499fb44ae0bed5a84fe1" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:11.881844 7950 consensus_queue.cc:1035] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad2d7819b135414d8ac89058eb71e7e2" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:11.890918 7959 mvcc.cc:205] Tried to move back new op lower bound from 7332848074255765504 to 7332848074069737472. Current Snapshot: MvccSnapshot[applied={T|T < 7332848074255765504}]
I20260924 10:35:11.892439 7958 mvcc.cc:205] Tried to move back new op lower bound from 7332848074255765504 to 7332848074069737472. Current Snapshot: MvccSnapshot[applied={T|T < 7332848074255765504}]
I20260924 10:35:11.898293 7475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:11.898439 7475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:11.900765 7475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b29c9e2f06d94efdbe78be35a8f6606d: Sending DeleteTablet for 3 replicas of tablet e0c975cc02fa4f42be6324b96acc3e52
I20260924 10:35:11.901654 7598 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c975cc02fa4f42be6324b96acc3e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:11 UTC) from {username='slave'} at 127.0.0.1:53168
I20260924 10:35:11.901654 7729 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c975cc02fa4f42be6324b96acc3e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:11 UTC) from {username='slave'} at 127.0.0.1:40594
I20260924 10:35:11.901721 7860 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c975cc02fa4f42be6324b96acc3e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:11 UTC) from {username='slave'} at 127.0.0.1:42236
I20260924 10:35:11.902161 7968 tablet_replica.cc:330] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: stopping tablet replica
I20260924 10:35:11.902383 7969 tablet_replica.cc:330] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: stopping tablet replica
I20260924 10:35:11.902485 7968 raft_consensus.cc:2229] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:11.902781 7967 tablet_replica.cc:330] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840: stopping tablet replica
I20260924 10:35:11.903000 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7536
I20260924 10:35:11.903064 7967 raft_consensus.cc:2229] T e0c975cc02fa4f42be6324b96acc3e52 P 0869ed4978e44fff87952d3241d56840 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:11.903769 7968 raft_consensus.cc:2258] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:11.905059 7969 raft_consensus.cc:2229] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:11.905370 7969 raft_consensus.cc:2258] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:11.905478 7968 ts_tablet_manager.cc:1905] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:11.907884 7968 ts_tablet_manager.cc:1918] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:11.907966 7968 log.cc:1196] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/e0c975cc02fa4f42be6324b96acc3e52
I20260924 10:35:11.908186 7968 ts_tablet_manager.cc:1939] T e0c975cc02fa4f42be6324b96acc3e52 P ad2d7819b135414d8ac89058eb71e7e2: Deleting consensus metadata
I20260924 10:35:11.909587 7463 catalog_manager.cc:4841] TS ad2d7819b135414d8ac89058eb71e7e2 (127.2.193.130:38685): tablet e0c975cc02fa4f42be6324b96acc3e52 (table test-table [id=8c467567136e4fc4924729cf70497f3f]) successfully deleted
I20260924 10:35:11.910332 7969 ts_tablet_manager.cc:1905] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:35:11.910946 7462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:43865 (error 108)
I20260924 10:35:11.911010 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7667
I20260924 10:35:11.912500 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:11.912116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:43865 (local address 127.0.0.1:53178)
0924 10:35:11.912315 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:11.912395 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 10:35:11.912760 7462 catalog_manager.cc:4584] TS 0869ed4978e44fff87952d3241d56840 (127.2.193.129:43865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0c975cc02fa4f42be6324b96acc3e52: Network error: Client connection negotiation failed: client connection to 127.2.193.129:43865: connect: Connection refused (error 111)
I20260924 10:35:11.913625 7969 ts_tablet_manager.cc:1918] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:11.913786 7969 log.cc:1196] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/e0c975cc02fa4f42be6324b96acc3e52
I20260924 10:35:11.914137 7969 ts_tablet_manager.cc:1939] T e0c975cc02fa4f42be6324b96acc3e52 P eda75675fb3e499fb44ae0bed5a84fe1: Deleting consensus metadata
I20260924 10:35:11.915313 7460 catalog_manager.cc:4841] TS eda75675fb3e499fb44ae0bed5a84fe1 (127.2.193.131:41769): tablet e0c975cc02fa4f42be6324b96acc3e52 (table test-table [id=8c467567136e4fc4924729cf70497f3f]) successfully deleted
I20260924 10:35:11.917448 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7798
I20260924 10:35:11.924096 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7445
2026-09-24T10:35:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1127 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 10:35:11.945509 2822 mini_oidc.cc:152] Starting JWKS server
I20260924 10:35:11.947103 2822 webserver.cc:460] Webserver started at https://127.0.0.1:38557/ using document root <none> and password file <none>
I20260924 10:35:11.947305 2822 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:35:11.947579 2822 webserver.cc:460] Webserver started at http://127.0.0.1:45861/ using document root <none> and password file <none>
2026-09-24T10:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:11Z Disabled control of system clock
I20260924 10:35:11.956454 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40533
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40533
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38557/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/testchainca.cert with env {}
W20260924 10:35:12.064669 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:12.064925 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:12.064965 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:12.068974 7976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:12.069031 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:12.069046 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:12.069062 7976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:12.069077 7976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:35:12.069093 7976 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:35:12.069106 7976 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38557/jwks.json
I20260924 10:35:12.074400 7976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38557/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:12.075345 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:12.076265 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:12.084712 7976 server_base.cc:1034] running on GCE node
W20260924 10:35:12.084753 7984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:12.084771 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:12.084873 7982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:12.085474 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:12.086570 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:12.087741 7976 hybrid_clock.cc:648] HybridClock initialized: now 1790246112087711 us; error 44 us; skew 500 ppm
I20260924 10:35:12.089651 7976 webserver.cc:460] Webserver started at http://127.2.193.190:33475/ using document root <none> and password file <none>
I20260924 10:35:12.090207 7976 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:12.090294 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:12.090513 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:12.092550 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "12043ec0e85e4bbfa7861c6f025e8831"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "2320c60199c1fb2f662a666ade0bd404"
server_key_iv: "ea9e710509853cfe38104087065b5793"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.093113 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "12043ec0e85e4bbfa7861c6f025e8831"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "2320c60199c1fb2f662a666ade0bd404"
server_key_iv: "ea9e710509853cfe38104087065b5793"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.096694 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:35:12.099087 7990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:12.100315 7976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:12.100451 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "12043ec0e85e4bbfa7861c6f025e8831"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "2320c60199c1fb2f662a666ade0bd404"
server_key_iv: "ea9e710509853cfe38104087065b5793"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.100559 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:12.119863 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:12.120551 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:12.120739 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:12.127525 7976 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:40533
I20260924 10:35:12.127537 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:40533 every 8 connection(s)
I20260924 10:35:12.128469 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:12.131618 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:12.131933 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 7976
I20260924 10:35:12.132068 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:12.132333 2822 external_mini_cluster.cc:1428] Setting key 090aec2bb3ebd1054c004c40f421fe2e
I20260924 10:35:12.139648 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831: Bootstrap starting.
I20260924 10:35:12.142521 8043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:12.143450 8043 log.cc:824] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:12.143507 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:12.133845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.2.193.190:40533)
0924 10:35:12.134544 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:12.134567 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:12.134614 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:35:12.135473 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:12.135500 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:12.136071 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:12.136476 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:12.136961 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.138617 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.139548 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.140279 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.140448 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.141285 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:12.141292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:12.141309 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:12.141843 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:12.141919 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:12.141947 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:12.142051 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:12.142180 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 10:35:12.142484 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20260924 10:35:12.145959 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831: No bootstrap required, opened a new log
I20260924 10:35:12.146549 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:35:12.150080 8043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12043ec0e85e4bbfa7861c6f025e8831" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40533 } }
I20260924 10:35:12.150318 8043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:12.150390 8043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12043ec0e85e4bbfa7861c6f025e8831, State: Initialized, Role: FOLLOWER
I20260924 10:35:12.150801 8043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12043ec0e85e4bbfa7861c6f025e8831" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40533 } }
I20260924 10:35:12.150939 8043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:12.151017 8043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:12.151126 8043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:12.152014 8043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12043ec0e85e4bbfa7861c6f025e8831" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40533 } }
I20260924 10:35:12.152343 8043 leader_election.cc:302] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12043ec0e85e4bbfa7861c6f025e8831; no voters:
I20260924 10:35:12.152678 8043 leader_election.cc:288] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:12.152796 8048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:12.153016 8048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [term 1 LEADER]: Becoming Leader. State: Replica: 12043ec0e85e4bbfa7861c6f025e8831, State: Running, Role: LEADER
I20260924 10:35:12.153321 8048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12043ec0e85e4bbfa7861c6f025e8831" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40533 } }
I20260924 10:35:12.153895 8043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:12.154786 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12043ec0e85e4bbfa7861c6f025e8831. Latest consensus state: current_term: 1 leader_uuid: "12043ec0e85e4bbfa7861c6f025e8831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12043ec0e85e4bbfa7861c6f025e8831" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40533 } } }
I20260924 10:35:12.154923 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:12.155211 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12043ec0e85e4bbfa7861c6f025e8831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12043ec0e85e4bbfa7861c6f025e8831" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40533 } } }
I20260924 10:35:12.155299 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:12.155438 8057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:12.158555 8057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:12.164188 8057 catalog_manager.cc:1347] Generated new cluster ID: b6324b95e0de4a2792f6c815e6e5da3d
I20260924 10:35:12.164270 8057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:12.176661 8057 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:12.177748 8057 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:12.191499 8057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12043ec0e85e4bbfa7861c6f025e8831: Generated new TSK 0
I20260924 10:35:12.192247 8057 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:35:12.231073 8073 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 10:35:12.231204 8073 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:35:12.231690 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:12.207384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.2.193.190:40533)
0924 10:35:12.207519 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:12.207524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:12.207751 (+ 227us) server_negotiation.cc:407] Connection header received
0924 10:35:12.207971 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:12.207973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:12.208020 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:12.208109 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:35:12.208426 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.208969 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.210228 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.210479 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.210564 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.210938 (+ 374us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:35:12.231218 (+ 20280us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:35:12.231574 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 10:35:12.231797 8046 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 10:35:12.233160 8072 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:12.207318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40533 (local address 127.0.0.1:42920)
0924 10:35:12.207659 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:12.207675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:12.207824 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:12.208143 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:12.208146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:12.208154 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:35:12.208279 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:12.208286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.209088 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.209092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:12.210110 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:12.210120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.210597 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.210601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:12.210701 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.210753 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:35:12.231548 (+ 20795us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:35:12.231832 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.193.190:40533: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":109,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.193.190:40533: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38557/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 10:35:12.234009 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 7976
2026-09-24T10:35:12Z chronyd exiting
I20260924 10:35:12.271533 2822 test_util.cc:175] -----------------------------------------------
I20260924 10:35:12.271627 2822 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246085895225-2822-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 10:35:12.272696 2822 mini_oidc.cc:152] Starting JWKS server
I20260924 10:35:12.273655 2822 webserver.cc:460] Webserver started at https://127.0.0.1:36681/ using document root <none> and password file <none>
I20260924 10:35:12.273761 2822 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:35:12.274037 2822 webserver.cc:460] Webserver started at http://127.0.0.1:41163/ using document root <none> and password file <none>
2026-09-24T10:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:12Z Disabled control of system clock
I20260924 10:35:12.282941 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:45813
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:45813
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36681/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/testchainca.cert with env {}
W20260924 10:35:12.390828 8079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:12.391096 8079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:12.391137 8079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:12.395130 8079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:12.395191 8079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:12.395207 8079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:12.395224 8079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:12.395239 8079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:35:12.395255 8079 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:35:12.395269 8079 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36681/jwks.json
I20260924 10:35:12.400601 8079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:45813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:45813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36681/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:12.401666 8079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:12.402633 8079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:12.410928 8079 server_base.cc:1034] running on GCE node
W20260924 10:35:12.410861 8085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:12.410877 8087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:12.410876 8084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:12.411703 8079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:12.412753 8079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:12.413933 8079 hybrid_clock.cc:648] HybridClock initialized: now 1790246112413901 us; error 44 us; skew 500 ppm
I20260924 10:35:12.415952 8079 webserver.cc:460] Webserver started at http://127.2.193.190:44509/ using document root <none> and password file <none>
I20260924 10:35:12.416468 8079 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:12.416527 8079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:12.416682 8079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:12.418686 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "0edc5b57051244fda5fc694460342afa"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "fc6f162dcda9956d08e9e5a634a05390"
server_key_iv: "a6061b4ae05b247a8c0963c1af90e6f5"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.419193 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "0edc5b57051244fda5fc694460342afa"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "fc6f162dcda9956d08e9e5a634a05390"
server_key_iv: "a6061b4ae05b247a8c0963c1af90e6f5"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.422616 8079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 10:35:12.424804 8093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:12.426071 8079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:12.426203 8079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "0edc5b57051244fda5fc694460342afa"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "fc6f162dcda9956d08e9e5a634a05390"
server_key_iv: "a6061b4ae05b247a8c0963c1af90e6f5"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.426311 8079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:12.462206 8079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:12.462963 8079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:12.463171 8079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:12.470163 8079 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:45813
I20260924 10:35:12.470172 8145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:45813 every 8 connection(s)
I20260924 10:35:12.471135 8079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:12.474460 8146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:12.479074 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8079
I20260924 10:35:12.479202 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:12.479487 2822 external_mini_cluster.cc:1428] Setting key d6453c07e783bf4722c3cf8c1e8a79ba
I20260924 10:35:12.480185 8146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa: Bootstrap starting.
I20260924 10:35:12.482468 8146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:12.483165 8146 log.cc:824] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:12.484895 8146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa: No bootstrap required, opened a new log
I20260924 10:35:12.488315 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:12.480806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.2.193.190:45813)
0924 10:35:12.481446 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:12.481462 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:12.481497 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:35:12.482100 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:12.482116 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:12.482490 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:12.482845 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:12.483299 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.484239 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.485133 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.485637 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.485813 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.486660 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:12.486665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:12.486678 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:12.487070 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:12.487122 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:12.487141 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:12.487245 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:12.487390 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 10:35:12.487627 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260924 10:35:12.488601 8146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edc5b57051244fda5fc694460342afa" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45813 } }
I20260924 10:35:12.488821 8146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:12.488900 8146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0edc5b57051244fda5fc694460342afa, State: Initialized, Role: FOLLOWER
I20260924 10:35:12.489382 8146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edc5b57051244fda5fc694460342afa" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45813 } }
I20260924 10:35:12.489529 8146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:12.489609 8146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:12.489703 8146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:12.490694 8146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edc5b57051244fda5fc694460342afa" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45813 } }
I20260924 10:35:12.491036 8146 leader_election.cc:302] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0edc5b57051244fda5fc694460342afa; no voters:
I20260924 10:35:12.491096 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:35:12.491369 8146 leader_election.cc:288] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:12.491431 8151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:12.491643 8151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [term 1 LEADER]: Becoming Leader. State: Replica: 0edc5b57051244fda5fc694460342afa, State: Running, Role: LEADER
I20260924 10:35:12.491938 8151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edc5b57051244fda5fc694460342afa" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45813 } }
I20260924 10:35:12.492614 8146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:12.493666 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0edc5b57051244fda5fc694460342afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edc5b57051244fda5fc694460342afa" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45813 } } }
I20260924 10:35:12.493701 8152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0edc5b57051244fda5fc694460342afa. Latest consensus state: current_term: 1 leader_uuid: "0edc5b57051244fda5fc694460342afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edc5b57051244fda5fc694460342afa" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45813 } } }
I20260924 10:35:12.493783 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:12.493806 8152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:12.494122 8160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:12.496809 8160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:12.501961 8160 catalog_manager.cc:1347] Generated new cluster ID: 50ba91f20f6d49cfa7a5c9bc87f9ccfd
I20260924 10:35:12.502035 8160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:12.523795 8160 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:12.524756 8160 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:12.532106 8160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0edc5b57051244fda5fc694460342afa: Generated new TSK 0
I20260924 10:35:12.532918 8160 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:35:12.565742 8176 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 10:35:12.565840 8176 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:35:12.566488 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:12.550884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.2.193.190:45813)
0924 10:35:12.551020 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:12.551027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:12.551190 (+ 163us) server_negotiation.cc:407] Connection header received
0924 10:35:12.551350 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:12.551353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:12.551400 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:12.551491 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:35:12.551733 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.552259 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.553488 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.553699 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.553778 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.554095 (+ 317us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:35:12.566149 (+ 12054us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:35:12.566379 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 10:35:12.566574 8149 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 10:35:12.566761 8175 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:12.550797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:45813 (local address 127.0.0.1:37670)
0924 10:35:12.551101 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:12.551116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:12.551231 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:12.551510 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:12.551513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:12.551521 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:35:12.551628 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:12.551634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.552362 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.552366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:12.553380 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:12.553390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.553809 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.553813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:12.553907 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.553935 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:35:12.566366 (+ 12431us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:35:12.566569 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.193.190:45813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36681/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
I20260924 10:35:12.567369 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8079
2026-09-24T10:35:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 10:35:12.606650 2822 mini_oidc.cc:152] Starting JWKS server
I20260924 10:35:12.607517 2822 webserver.cc:460] Webserver started at https://127.0.0.1:40881/ using document root <none> and password file <none>
I20260924 10:35:12.607601 2822 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:35:12.607817 2822 webserver.cc:460] Webserver started at http://127.0.0.1:35217/ using document root <none> and password file <none>
2026-09-24T10:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:12Z Disabled control of system clock
I20260924 10:35:12.616510 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:38627
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:43877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:38627
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40881/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:35:12.730022 8182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:12.730326 8182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:12.730367 8182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:12.734614 8182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:12.734686 8182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:12.734702 8182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:12.734719 8182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:12.734735 8182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:35:12.734751 8182 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:35:12.734764 8182 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40881/jwks.json
I20260924 10:35:12.739728 8182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:43877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:38627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40881/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:12.740849 8182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:12.742249 8182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:12.751276 8190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:12.751266 8188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:12.751318 8182 server_base.cc:1034] running on GCE node
W20260924 10:35:12.751266 8187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:12.752135 8182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:12.753433 8182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:12.754628 8182 hybrid_clock.cc:648] HybridClock initialized: now 1790246112754594 us; error 46 us; skew 500 ppm
I20260924 10:35:12.756613 8182 webserver.cc:460] Webserver started at http://127.2.193.190:43663/ using document root <none> and password file <none>
I20260924 10:35:12.757341 8182 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:12.757441 8182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:12.757669 8182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:12.759855 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "c17ae5374b7c4d7f8258bffafcedf1b3"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "e64336ea20928962224306124d0d7ee7"
server_key_iv: "2a107be12a2d97633b87823fe1835cdf"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.760447 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "c17ae5374b7c4d7f8258bffafcedf1b3"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "e64336ea20928962224306124d0d7ee7"
server_key_iv: "2a107be12a2d97633b87823fe1835cdf"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.764458 8182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:12.767066 8196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:12.768391 8182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:12.768539 8182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "c17ae5374b7c4d7f8258bffafcedf1b3"
format_stamp: "Formatted at 2026-09-24 10:35:12 on dist-test-slave-8wc7"
server_key: "e64336ea20928962224306124d0d7ee7"
server_key_iv: "2a107be12a2d97633b87823fe1835cdf"
server_key_version: "encryptionkey@0"
I20260924 10:35:12.768657 8182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:12.780030 8182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:12.780788 8182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:12.780992 8182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:12.788091 8182 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:38627
I20260924 10:35:12.788105 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:38627 every 8 connection(s)
I20260924 10:35:12.789090 8182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:12.792188 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8182
I20260924 10:35:12.792258 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:12.792351 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:12.792639 2822 external_mini_cluster.cc:1428] Setting key cc691cc00ab8a34808692c38672754cd
I20260924 10:35:12.799415 8249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3: Bootstrap starting.
I20260924 10:35:12.802166 8249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:12.802783 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:12.794025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.2.193.190:38627)
0924 10:35:12.794650 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:12.794666 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:12.794703 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:35:12.795619 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:12.795640 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:12.796067 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:12.796377 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:12.796827 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.797933 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.799157 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.799754 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.799884 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.800797 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:12.800802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:12.800816 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:12.801302 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:12.801362 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:12.801384 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:12.801472 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:12.801622 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:35:12.801931 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20260924 10:35:12.802981 8249 log.cc:824] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:12.805366 8249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3: No bootstrap required, opened a new log
I20260924 10:35:12.805989 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:35:12.809319 8249 raft_consensus.cc:348] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38627 } }
I20260924 10:35:12.809557 8249 raft_consensus.cc:374] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:12.809597 8249 raft_consensus.cc:729] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17ae5374b7c4d7f8258bffafcedf1b3, State: Initialized, Role: FOLLOWER
I20260924 10:35:12.810012 8249 consensus_queue.cc:260] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38627 } }
I20260924 10:35:12.810132 8249 raft_consensus.cc:388] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:12.810177 8249 raft_consensus.cc:482] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:12.810251 8249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:12.811178 8249 raft_consensus.cc:504] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38627 } }
I20260924 10:35:12.811481 8249 leader_election.cc:302] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c17ae5374b7c4d7f8258bffafcedf1b3; no voters:
I20260924 10:35:12.811761 8249 leader_election.cc:288] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:12.812036 8254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:12.812266 8254 raft_consensus.cc:686] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [term 1 LEADER]: Becoming Leader. State: Replica: c17ae5374b7c4d7f8258bffafcedf1b3, State: Running, Role: LEADER
I20260924 10:35:12.812577 8254 consensus_queue.cc:237] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38627 } }
I20260924 10:35:12.813078 8249 sys_catalog.cc:564] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:12.814550 8255 sys_catalog.cc:455] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38627 } } }
I20260924 10:35:12.814685 8255 sys_catalog.cc:458] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:12.814970 8256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c17ae5374b7c4d7f8258bffafcedf1b3. Latest consensus state: current_term: 1 leader_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c17ae5374b7c4d7f8258bffafcedf1b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38627 } } }
I20260924 10:35:12.815058 8263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:12.815079 8256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:12.818063 8263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:12.823441 8263 catalog_manager.cc:1347] Generated new cluster ID: 4811ee9ec2c7468e8ed9e6e2d4888fdc
I20260924 10:35:12.823539 8263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:12.839751 8263 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:12.840718 8263 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:12.845613 8263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c17ae5374b7c4d7f8258bffafcedf1b3: Generated new TSK 0
I20260924 10:35:12.846313 8263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:35:12.865360 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:12.860299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.2.193.190:38627)
0924 10:35:12.860433 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:12.860437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:12.860586 (+ 149us) server_negotiation.cc:407] Connection header received
0924 10:35:12.860747 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:12.860749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:12.860796 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:12.860898 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:35:12.861159 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.861738 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.862981 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.863197 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.863269 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.863634 (+ 365us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:35:12.864917 (+ 1283us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:35:12.865180 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52780: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 10:35:12.865451 8252 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52780: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 10:35:12.868299 8278 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:12.860149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:38627 (local address 127.0.0.1:52780)
0924 10:35:12.860490 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:12.860506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:12.860624 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:12.860926 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:12.860929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:12.860936 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:35:12.861057 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:12.861063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.861852 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.861856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:12.862850 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:12.862860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:12.863307 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:12.863311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:12.863399 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:12.863429 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:35:12.865089 (+ 1660us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:35:12.868140 (+ 3051us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.193.190:38627: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":2842,"thread_start_us":108,"threads_started":1}
I20260924 10:35:12.869010 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8182
E20260924 10:35:12.879452 8279 webserver.cc:533] Webserver: sslize failed:
E20260924 10:35:12.879544 8279 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T10:35:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T10:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:12Z Disabled control of system clock
I20260924 10:35:12.917599 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:35707
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:35707
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/testchainca.cert with env {}
W20260924 10:35:13.032986 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:13.033456 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:13.033543 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:13.037731 8283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:13.037842 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:13.037873 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:13.037914 8283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:13.037947 8283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:35:13.037981 8283 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:35:13.038012 8283 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 10:35:13.043926 8283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:35707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:13.045090 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:13.046202 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:13.055096 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:13.055078 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:13.055083 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:13.055215 8283 server_base.cc:1034] running on GCE node
I20260924 10:35:13.055961 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:13.057116 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:13.058333 8283 hybrid_clock.cc:648] HybridClock initialized: now 1790246113058291 us; error 54 us; skew 500 ppm
I20260924 10:35:13.060279 8283 webserver.cc:460] Webserver started at http://127.2.193.190:40967/ using document root <none> and password file <none>
I20260924 10:35:13.060889 8283 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:13.060981 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:13.061259 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:13.063438 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "56b2d50772874852ad45bb6f62a84913"
format_stamp: "Formatted at 2026-09-24 10:35:13 on dist-test-slave-8wc7"
server_key: "537cdf248bfed6a057df769e2a4e8329"
server_key_iv: "d1631ed97dacb84f7cfea2a65da5ac13"
server_key_version: "encryptionkey@0"
I20260924 10:35:13.064038 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "56b2d50772874852ad45bb6f62a84913"
format_stamp: "Formatted at 2026-09-24 10:35:13 on dist-test-slave-8wc7"
server_key: "537cdf248bfed6a057df769e2a4e8329"
server_key_iv: "d1631ed97dacb84f7cfea2a65da5ac13"
server_key_version: "encryptionkey@0"
I20260924 10:35:13.067883 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:13.070477 8297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:13.071794 8283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:13.071956 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "56b2d50772874852ad45bb6f62a84913"
format_stamp: "Formatted at 2026-09-24 10:35:13 on dist-test-slave-8wc7"
server_key: "537cdf248bfed6a057df769e2a4e8329"
server_key_iv: "d1631ed97dacb84f7cfea2a65da5ac13"
server_key_version: "encryptionkey@0"
I20260924 10:35:13.072078 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:13.086643 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:13.087397 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:13.087602 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:13.094686 8283 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:35707
I20260924 10:35:13.094756 8349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:35707 every 8 connection(s)
I20260924 10:35:13.095724 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:13.099086 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:13.103426 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8283
I20260924 10:35:13.103576 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:13.103868 2822 external_mini_cluster.cc:1428] Setting key 7956f50ea1d4fc8a7df55cb40064a903
I20260924 10:35:13.105350 8350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913: Bootstrap starting.
I20260924 10:35:13.107632 8350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:13.108387 8350 log.cc:824] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:13.110308 8350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913: No bootstrap required, opened a new log
I20260924 10:35:13.113456 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:13.105393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.2.193.190:35707)
0924 10:35:13.106041 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:13.106057 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:13.106094 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:35:13.106733 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:13.106749 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:13.107154 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:13.107524 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:13.108018 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:13.109265 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:13.110136 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:13.110662 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:13.110833 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:13.111683 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:13.111688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:13.111700 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:13.112090 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:13.112142 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:13.112163 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:13.112271 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:13.112413 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 10:35:13.112671 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260924 10:35:13.114215 8350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b2d50772874852ad45bb6f62a84913" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35707 } }
I20260924 10:35:13.114482 8350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:13.114545 8350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b2d50772874852ad45bb6f62a84913, State: Initialized, Role: FOLLOWER
I20260924 10:35:13.115025 8350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b2d50772874852ad45bb6f62a84913" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35707 } }
I20260924 10:35:13.115171 8350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:13.115235 8350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:13.115339 8350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:13.116420 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:35:13.116410 8350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b2d50772874852ad45bb6f62a84913" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35707 } }
I20260924 10:35:13.116737 8350 leader_election.cc:302] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56b2d50772874852ad45bb6f62a84913; no voters:
I20260924 10:35:13.117044 8350 leader_election.cc:288] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:13.117153 8355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:13.117389 8355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [term 1 LEADER]: Becoming Leader. State: Replica: 56b2d50772874852ad45bb6f62a84913, State: Running, Role: LEADER
I20260924 10:35:13.117707 8355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b2d50772874852ad45bb6f62a84913" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35707 } }
I20260924 10:35:13.118412 8350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:13.119495 8357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56b2d50772874852ad45bb6f62a84913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b2d50772874852ad45bb6f62a84913" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35707 } } }
I20260924 10:35:13.119637 8357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:13.119887 8356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56b2d50772874852ad45bb6f62a84913. Latest consensus state: current_term: 1 leader_uuid: "56b2d50772874852ad45bb6f62a84913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b2d50772874852ad45bb6f62a84913" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35707 } } }
I20260924 10:35:13.119994 8356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913 [sys.catalog]: This master's current role is: LEADER
W20260924 10:35:13.123241 8370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:35:13.123335 8370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:35:13.123404 8364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:13.124317 8364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:13.130019 8364 catalog_manager.cc:1347] Generated new cluster ID: 7d3678e29dec4b49b2c8a7815df6128b
I20260924 10:35:13.130120 8364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:13.154071 8364 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:13.155081 8364 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:13.160041 8364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56b2d50772874852ad45bb6f62a84913: Generated new TSK 0
I20260924 10:35:13.160794 8364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:35:14.035248 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:13.174363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.2.193.190:35707)
0924 10:35:13.174496 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:13.174501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:13.174709 (+ 208us) server_negotiation.cc:407] Connection header received
0924 10:35:13.174922 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:13.174924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:13.174973 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:13.175092 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:35:13.175382 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:13.176069 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:13.177144 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:13.177420 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:13.177537 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:13.177865 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:35:14.034791 (+856926us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:35:14.035113 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57242: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:35:14.035386 8353 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57242: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 10:35:14.035401 8379 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:13.174255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:35707 (local address 127.0.0.1:57242)
0924 10:35:13.174616 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:13.174629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:13.174765 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:13.175087 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:13.175089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:13.175097 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:35:13.175216 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:13.175222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:13.176183 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:13.176187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:13.177018 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:13.177027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:13.177548 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:13.177552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:13.177647 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:13.177678 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:35:14.035054 (+857376us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:35:14.035246 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.193.190:35707: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260924 10:35:14.036173 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8283
2026-09-24T10:35:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1147 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (129 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:14 dist-test-slave-8wc7 krb5kdc[8391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:14 dist-test-slave-8wc7 krb5kdc[8391](info): set up 2 sockets
Sep 24 10:35:14 dist-test-slave-8wc7 krb5kdc[8391](info): commencing operation
krb5kdc: starting...
W20260924 10:35:15.321478 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:15 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:15.440522 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34185
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:34185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:15.546324 8404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:15.546625 8404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:15.546698 8404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:15.550734 8404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:15.550820 8404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:15.550870 8404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:15.550921 8404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:15.550961 8404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:15.556051 8404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:34185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:15.557085 8404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:15.558101 8404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:15.566510 8412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:15.566510 8409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:15.566519 8410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:15.566793 8404 server_base.cc:1034] running on GCE node
I20260924 10:35:15.567385 8404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:15.568388 8404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:15.569576 8404 hybrid_clock.cc:648] HybridClock initialized: now 1790246115569565 us; error 28 us; skew 500 ppm
Sep 24 10:35:15 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:15.575578 8404 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:15.576395 8404 webserver.cc:460] Webserver started at http://127.2.193.190:35367/ using document root <none> and password file <none>
I20260924 10:35:15.576897 8404 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:15.576960 8404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:15.577131 8404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:15.579180 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "24c47e4585a14fc6b53e04e4aefc87c2"
format_stamp: "Formatted at 2026-09-24 10:35:15 on dist-test-slave-8wc7"
server_key: "a93904912a8501d34d8c1fbfdfed511b"
server_key_iv: "662a8cab12559380e987b418f7e3cd5e"
server_key_version: "encryptionkey@0"
I20260924 10:35:15.579690 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "24c47e4585a14fc6b53e04e4aefc87c2"
format_stamp: "Formatted at 2026-09-24 10:35:15 on dist-test-slave-8wc7"
server_key: "a93904912a8501d34d8c1fbfdfed511b"
server_key_iv: "662a8cab12559380e987b418f7e3cd5e"
server_key_version: "encryptionkey@0"
I20260924 10:35:15.583400 8404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:35:15.585714 8419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:15.586905 8404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:15.587016 8404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "24c47e4585a14fc6b53e04e4aefc87c2"
format_stamp: "Formatted at 2026-09-24 10:35:15 on dist-test-slave-8wc7"
server_key: "a93904912a8501d34d8c1fbfdfed511b"
server_key_iv: "662a8cab12559380e987b418f7e3cd5e"
server_key_version: "encryptionkey@0"
I20260924 10:35:15.587105 8404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:15.630700 8404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:15.633613 8404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:15.633769 8404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:15.640345 8404 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:34185
I20260924 10:35:15.640360 8471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:34185 every 8 connection(s)
I20260924 10:35:15.641341 8404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:15.644034 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:15.646406 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8404
I20260924 10:35:15.646512 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:15.646785 2822 external_mini_cluster.cc:1428] Setting key 83132ebb00af2bf967a63595f5c77b31
I20260924 10:35:15.650175 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2: Bootstrap starting.
I20260924 10:35:15.652678 8472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:15.653487 8472 log.cc:824] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:15.655372 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2: No bootstrap required, opened a new log
Sep 24 10:35:15 dist-test-slave-8wc7 krb5kdc[8391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:15.659282 8472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34185 } }
I20260924 10:35:15.659529 8472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:15.659579 8472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c47e4585a14fc6b53e04e4aefc87c2, State: Initialized, Role: FOLLOWER
I20260924 10:35:15.660049 8472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34185 } }
I20260924 10:35:15.660204 8472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:15.660269 8472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:15.660353 8472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:15.661405 8472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34185 } }
I20260924 10:35:15.661743 8472 leader_election.cc:302] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24c47e4585a14fc6b53e04e4aefc87c2; no voters:
I20260924 10:35:15.662060 8472 leader_election.cc:288] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:15.662163 8477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:15.662165 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:15.648178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.2.193.190:34185)
0924 10:35:15.648567 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:15.648585 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:15.648625 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:35:15.649489 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:15.649511 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:15.649918 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:15.650268 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:15.650965 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:15.652415 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:15.653296 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:15.653961 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:15.654077 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:15.656583 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:15.656589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:15.656605 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:15.656632 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:15.659140 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:15.659540 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:15.659548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:15.659554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:15.659689 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:15.659975 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:15.659978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:15.659980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:15.660494 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:15.660699 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:15.660893 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:35:15.661206 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
I20260924 10:35:15.662402 8477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [term 1 LEADER]: Becoming Leader. State: Replica: 24c47e4585a14fc6b53e04e4aefc87c2, State: Running, Role: LEADER
I20260924 10:35:15.662722 8477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34185 } }
I20260924 10:35:15.663209 8472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:15.665051 8479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c47e4585a14fc6b53e04e4aefc87c2. Latest consensus state: current_term: 1 leader_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34185 } } }
I20260924 10:35:15.665160 8479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:15.665514 8478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c47e4585a14fc6b53e04e4aefc87c2" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34185 } } }
I20260924 10:35:15.665592 8491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:15.665611 8478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:15.668172 8491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:15.673472 8491 catalog_manager.cc:1347] Generated new cluster ID: a045e8dc3e334718a5968ac4a029a99e
I20260924 10:35:15.673537 8491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:15.693104 8491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:15.694339 8491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:15.706833 8491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c47e4585a14fc6b53e04e4aefc87c2: Generated new TSK 0
I20260924 10:35:15.707653 8491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:35:15.743887 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34185
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:15.852548 8498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:15.852806 8498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:15.852849 8498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:15.856792 8498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:15.856845 8498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:15.856957 8498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:15.862136 8498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:15.863153 8498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:15.864228 8498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:15.873986 8506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:15.873982 8504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:15.873986 8503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:15.874137 8498 server_base.cc:1034] running on GCE node
I20260924 10:35:15.874723 8498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:15.875331 8498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:15.876488 8498 hybrid_clock.cc:648] HybridClock initialized: now 1790246115876472 us; error 28 us; skew 500 ppm
Sep 24 10:35:15 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:15.879235 8498 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:35:15.880093 8498 webserver.cc:460] Webserver started at http://127.2.193.129:35515/ using document root <none> and password file <none>
I20260924 10:35:15.880651 8498 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:15.880739 8498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:15.880957 8498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:15.883023 8498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "454f102e04a248728c12da0fa5951e92"
format_stamp: "Formatted at 2026-09-24 10:35:15 on dist-test-slave-8wc7"
server_key: "1704534bb6ede570d22e208f048835a6"
server_key_iv: "ae5de9a069aea4e1964dab09741a872f"
server_key_version: "encryptionkey@0"
I20260924 10:35:15.883589 8498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "454f102e04a248728c12da0fa5951e92"
format_stamp: "Formatted at 2026-09-24 10:35:15 on dist-test-slave-8wc7"
server_key: "1704534bb6ede570d22e208f048835a6"
server_key_iv: "ae5de9a069aea4e1964dab09741a872f"
server_key_version: "encryptionkey@0"
I20260924 10:35:15.887209 8498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:35:15.889544 8513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:15.890698 8498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:15.890832 8498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "454f102e04a248728c12da0fa5951e92"
format_stamp: "Formatted at 2026-09-24 10:35:15 on dist-test-slave-8wc7"
server_key: "1704534bb6ede570d22e208f048835a6"
server_key_iv: "ae5de9a069aea4e1964dab09741a872f"
server_key_version: "encryptionkey@0"
I20260924 10:35:15.890935 8498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:15.913317 8498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:15.916707 8498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:15.916909 8498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:15.917559 8498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:15.918344 8498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:15.918412 8498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:15.918507 8498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:15.918555 8498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:15.927045 8498 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:37905
I20260924 10:35:15.927070 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:37905 every 8 connection(s)
I20260924 10:35:15.927930 8498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:15.929913 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8498
I20260924 10:35:15.930038 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:15.930303 2822 external_mini_cluster.cc:1428] Setting key 3d2e79619cc7cf5af8040aa52ea21f8c
Sep 24 10:35:15 dist-test-slave-8wc7 krb5kdc[8391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:15.942924 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:15.929601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:53507 (local address 127.2.193.190:34185)
0924 10:35:15.929772 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:15.929777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:15.930671 (+ 894us) server_negotiation.cc:407] Connection header received
0924 10:35:15.932243 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:15.932249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:15.932310 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:15.932402 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:15.933728 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:15.934259 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:15.936069 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:15.936284 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:15.936364 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:15.939357 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:15.939360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:15.939362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:15.939387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:15.940800 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:15.941370 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:15.941373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:15.941374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:15.941446 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:15.941715 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:15.941717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:15.941718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:15.941875 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:15.941953 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:15.942559 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 10:35:15.942708 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:35:15.943885 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:15.930035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34185 (local address 127.2.193.129:53507)
0924 10:35:15.930521 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:15.930573 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:15.932029 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:15.932532 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:15.932544 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:15.932883 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:15.933568 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:15.933590 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:15.934390 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:15.934397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:15.935939 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:15.935954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:15.936440 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:15.936447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:15.936680 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:15.937375 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:15.937401 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:15.939187 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:15.940902 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:15.940907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:15.940919 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:15.941284 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:15.941522 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:15.941524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:15.941525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:15.941646 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:15.941984 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:15.941989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:15.942217 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 10:35:15.942478 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 10:35:15.942778 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
I20260924 10:35:15.945102 8627 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34185
I20260924 10:35:15.945372 8627 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:15.945842 8627 heartbeater.cc:511] Master 127.2.193.190:34185 requested a full tablet report, sending...
I20260924 10:35:15.947247 8436 ts_manager.cc:194] Registered new tserver with Master: 454f102e04a248728c12da0fa5951e92 (127.2.193.129:37905)
I20260924 10:35:15.948351 8436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:53507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:35:15.958217 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34185
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:16.067301 8632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:16.067588 8632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:16.067667 8632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:16.072006 8632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:16.072070 8632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:16.072131 8632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:16.077464 8632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:16.078537 8632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:16.079569 8632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:16.089123 8637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:16.089144 8640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:16.089144 8638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:16.089481 8632 server_base.cc:1034] running on GCE node
I20260924 10:35:16.089869 8632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:16.090507 8632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:16.091683 8632 hybrid_clock.cc:648] HybridClock initialized: now 1790246116091660 us; error 35 us; skew 500 ppm
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.094532 8632 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:35:16.095435 8632 webserver.cc:460] Webserver started at http://127.2.193.130:40387/ using document root <none> and password file <none>
I20260924 10:35:16.095996 8632 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:16.096056 8632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:16.096295 8632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:16.098429 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "52d079c1186844af893083c21bc708de"
format_stamp: "Formatted at 2026-09-24 10:35:16 on dist-test-slave-8wc7"
server_key: "ed8fcf1e7738605d39d884aee6f4d9bd"
server_key_iv: "328a32752084268e7b2f77ec84fa2077"
server_key_version: "encryptionkey@0"
I20260924 10:35:16.098990 8632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "52d079c1186844af893083c21bc708de"
format_stamp: "Formatted at 2026-09-24 10:35:16 on dist-test-slave-8wc7"
server_key: "ed8fcf1e7738605d39d884aee6f4d9bd"
server_key_iv: "328a32752084268e7b2f77ec84fa2077"
server_key_version: "encryptionkey@0"
I20260924 10:35:16.102804 8632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:35:16.105289 8647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:16.106468 8632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:16.106586 8632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "52d079c1186844af893083c21bc708de"
format_stamp: "Formatted at 2026-09-24 10:35:16 on dist-test-slave-8wc7"
server_key: "ed8fcf1e7738605d39d884aee6f4d9bd"
server_key_iv: "328a32752084268e7b2f77ec84fa2077"
server_key_version: "encryptionkey@0"
I20260924 10:35:16.106686 8632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:16.137140 8632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:16.140523 8632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:16.140728 8632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:16.141371 8632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:16.142215 8632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:16.142261 8632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:16.142325 8632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:16.142365 8632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:16.151120 8632 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:46639
I20260924 10:35:16.151146 8760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:46639 every 8 connection(s)
I20260924 10:35:16.152005 8632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:16.154618 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8632
I20260924 10:35:16.154727 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:16.155016 2822 external_mini_cluster.cc:1428] Setting key c7a5e5345d124a7713f2ae84ccdef397
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:16.167162 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:16.153702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:37829 (local address 127.2.193.190:34185)
0924 10:35:16.153822 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.153826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.154642 (+ 816us) server_negotiation.cc:407] Connection header received
0924 10:35:16.156153 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.156156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.156206 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.156290 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.157578 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.158340 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.159791 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.160015 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.160100 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:16.163290 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:16.163295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:16.163302 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:16.163336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:16.164823 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.165530 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.165538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.165539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.165635 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.166089 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.166092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.166094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.166323 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:16.166467 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:16.166934 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 10:35:16.167040 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:35:16.167964 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:16.154029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34185 (local address 127.2.193.130:37829)
0924 10:35:16.154504 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:16.154545 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:16.155952 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:16.156519 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:16.156529 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:16.156901 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:16.157443 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:16.157458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.158469 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.158473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:16.159667 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:16.159682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.160167 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.160174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:16.160413 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:16.161030 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:16.161052 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:16.163053 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:16.164943 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:16.164953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:16.164973 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:16.165394 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:16.165761 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:16.165767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:16.165771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:16.165961 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:16.166456 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:16.166462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:16.166706 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 10:35:16.166859 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 10:35:16.167117 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
I20260924 10:35:16.169193 8761 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34185
I20260924 10:35:16.169476 8761 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:16.170059 8761 heartbeater.cc:511] Master 127.2.193.190:34185 requested a full tablet report, sending...
I20260924 10:35:16.171160 8436 ts_manager.cc:194] Registered new tserver with Master: 52d079c1186844af893083c21bc708de (127.2.193.130:46639)
I20260924 10:35:16.172019 8436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:37829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:35:16.184129 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34185
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:16.296598 8766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:16.296865 8766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:16.296909 8766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:16.300917 8766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:16.300980 8766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:16.301038 8766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:16.306322 8766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:16.307356 8766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:16.308362 8766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:16.317848 8771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:16.318032 8766 server_base.cc:1034] running on GCE node
W20260924 10:35:16.317866 8772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:16.317862 8774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:16.318521 8766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:16.319151 8766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:16.320336 8766 hybrid_clock.cc:648] HybridClock initialized: now 1790246116320314 us; error 41 us; skew 500 ppm
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.323081 8766 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:35:16.323951 8766 webserver.cc:460] Webserver started at http://127.2.193.131:39131/ using document root <none> and password file <none>
I20260924 10:35:16.324501 8766 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:16.324589 8766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:16.324812 8766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:16.326886 8766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "4c67fc6d0d4141d6b11df67eba84346e"
format_stamp: "Formatted at 2026-09-24 10:35:16 on dist-test-slave-8wc7"
server_key: "ac8d644c1e5f39d356b770ddf1c1c1be"
server_key_iv: "70395335ce4de621a043a58567484f26"
server_key_version: "encryptionkey@0"
I20260924 10:35:16.327457 8766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "4c67fc6d0d4141d6b11df67eba84346e"
format_stamp: "Formatted at 2026-09-24 10:35:16 on dist-test-slave-8wc7"
server_key: "ac8d644c1e5f39d356b770ddf1c1c1be"
server_key_iv: "70395335ce4de621a043a58567484f26"
server_key_version: "encryptionkey@0"
I20260924 10:35:16.331113 8766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:35:16.333457 8781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:16.334618 8766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:16.334746 8766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "4c67fc6d0d4141d6b11df67eba84346e"
format_stamp: "Formatted at 2026-09-24 10:35:16 on dist-test-slave-8wc7"
server_key: "ac8d644c1e5f39d356b770ddf1c1c1be"
server_key_iv: "70395335ce4de621a043a58567484f26"
server_key_version: "encryptionkey@0"
I20260924 10:35:16.334853 8766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:16.354032 8766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:16.357079 8766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:16.357317 8766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:16.357924 8766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:16.358728 8766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:16.358796 8766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:16.358878 8766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:16.358932 8766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:16.367731 8766 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:43409
I20260924 10:35:16.367754 8894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:43409 every 8 connection(s)
I20260924 10:35:16.368603 8766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:16.370265 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8766
I20260924 10:35:16.370393 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:16.370716 2822 external_mini_cluster.cc:1428] Setting key 86a74e66347513f97c9d5af7dbebeb94
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:16.381676 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:16.370346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:45701 (local address 127.2.193.190:34185)
0924 10:35:16.370540 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.370546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.371389 (+ 843us) server_negotiation.cc:407] Connection header received
0924 10:35:16.372963 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.372967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.373021 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.373095 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.374224 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.374758 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.375869 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.376100 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.376206 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:16.378681 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:16.378684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:16.378686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:16.378709 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:16.379925 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.380366 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.380369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.380370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.380446 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.380724 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.380726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.380728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.380902 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:16.380974 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:16.381463 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 10:35:16.381560 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 10:35:16.382477 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:16.370740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34185 (local address 127.2.193.131:45701)
0924 10:35:16.371242 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:16.371297 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:16.372770 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:16.373272 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:16.373280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:16.373623 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:16.374090 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:16.374102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.374878 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.374881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:16.375749 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:16.375757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.376239 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.376242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:16.376406 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:16.376990 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:16.377006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:16.378536 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:16.380022 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:16.380027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:16.380038 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:16.380282 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:16.380519 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:16.380521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:16.380522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:16.380653 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:16.380984 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:16.380989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:16.381262 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 10:35:16.381420 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 10:35:16.381668 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
I20260924 10:35:16.383601 8895 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34185
I20260924 10:35:16.383846 8895 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:16.384387 8895 heartbeater.cc:511] Master 127.2.193.190:34185 requested a full tablet report, sending...
I20260924 10:35:16.385365 8436 ts_manager.cc:194] Registered new tserver with Master: 4c67fc6d0d4141d6b11df67eba84346e (127.2.193.131:43409)
I20260924 10:35:16.385912 8436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:45701
I20260924 10:35:16.396042 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/test-admin.keytab.
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.413839 2822 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 10:35:16.416724 8902 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:16.415209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.2.193.129:37905)
0924 10:35:16.415492 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.415509 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.415538 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:35:16.415651 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.415656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.415844 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.416003 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.416485 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46804: BlockingRecv error: recv got EOF from 127.0.0.1:46804 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
W20260924 10:35:16.416828 2822 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.417939 2822 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.418864 2822 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:35:16.427649 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:16.419682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.2.193.129:37905)
0924 10:35:16.419819 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.419822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.419836 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:16.419940 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.419942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.419979 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.420055 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.420509 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.421335 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.422109 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.422450 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.422530 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:16.424677 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:16.424686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:16.424699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:16.424721 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:16.426171 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.426553 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.426558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.426559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.426628 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.426832 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.426834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.426835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.427050 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:16.427213 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:16.427391 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 10:35:16.427513 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 10:35:16.431432 8902 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:16.430410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.2.193.129:37905)
0924 10:35:16.430536 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.430540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.430645 (+ 105us) server_negotiation.cc:407] Connection header received
0924 10:35:16.430777 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.430784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.430818 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.430894 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.431314 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46818: BlockingRecv error: recv got EOF from 127.0.0.1:46818 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 10:35:16.431586 2822 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.432555 2822 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.433454 2822 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:35:16.441815 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:16.434226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.2.193.129:37905)
0924 10:35:16.434348 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.434352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.434499 (+ 147us) server_negotiation.cc:407] Connection header received
0924 10:35:16.434627 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.434630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.434666 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.434747 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.435210 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.435733 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.436499 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.436733 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.436833 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:16.439024 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:16.439028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:16.439031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:16.439053 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:16.440242 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.440638 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.440642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.440644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.440719 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.440993 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.440996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.440998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.441152 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:16.441321 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:16.441549 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 10:35:16.441656 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 10:35:16.444429 8902 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:16.443498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.2.193.129:37905)
0924 10:35:16.443624 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.443628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.443740 (+ 112us) server_negotiation.cc:407] Connection header received
0924 10:35:16.443852 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.443855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.443890 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.443974 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.444329 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46836: BlockingRecv error: recv got EOF from 127.0.0.1:46836 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:35:16.444572 2822 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246085895225-2822-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.445556 2822 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:16.446385 2822 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:35:16.453992 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:16.447104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.2.193.129:37905)
0924 10:35:16.447226 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:16.447229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:16.447275 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:35:16.447368 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:16.447370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:16.447400 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:16.447464 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:16.447863 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.448328 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.449022 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:16.449285 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:16.449376 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:16.451340 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:16.451343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:16.451345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:16.451364 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:16.452661 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.452987 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.452989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.452990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.453055 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:16.453304 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:16.453306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:16.453308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:16.453465 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:16.453557 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:16.453716 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 10:35:16.453804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:35:16.454890 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8498
I20260924 10:35:16.460403 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8632
I20260924 10:35:16.465456 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8766
I20260924 10:35:16.471674 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8404
2026-09-24T10:35:16Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2181 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8915](info): set up 2 sockets
Sep 24 10:35:16 dist-test-slave-8wc7 krb5kdc[8915](info): commencing operation
krb5kdc: starting...
W20260924 10:35:17.509702 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:17 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab.
Entry for principal oryx/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab.
I20260924 10:35:17.632019 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40919
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab
--principal=oryx/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:17.745318 8928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:17.745679 8928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:17.745759 8928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:17.749907 8928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:17.750012 8928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:17.750051 8928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:17.750085 8928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:17.750116 8928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:17.755754 8928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab
--principal=oryx/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:17.756946 8928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:17.758044 8928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:17.766444 8934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:17.766438 8936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:17.766479 8933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:17.766782 8928 server_base.cc:1034] running on GCE node
I20260924 10:35:17.767377 8928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:17.768455 8928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:17.769609 8928 hybrid_clock.cc:648] HybridClock initialized: now 1790246117769590 us; error 32 us; skew 500 ppm
Sep 24 10:35:17 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246117, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:17.772253 8928 init.cc:377] Logged in from keytab as oryx/127.2.193.190@KRBTEST.COM (short username oryx)
I20260924 10:35:17.773204 8928 webserver.cc:460] Webserver started at http://127.2.193.190:41319/ using document root <none> and password file <none>
I20260924 10:35:17.773767 8928 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:17.773861 8928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:17.774094 8928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:17.776194 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "e3f037e6aa20415a9226a44178f67e35"
format_stamp: "Formatted at 2026-09-24 10:35:17 on dist-test-slave-8wc7"
server_key: "50eefed958fc6861f85a2186b29aadf9"
server_key_iv: "19d0ed696227292aceaa457027d697e9"
server_key_version: "encryptionkey@0"
I20260924 10:35:17.776806 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "e3f037e6aa20415a9226a44178f67e35"
format_stamp: "Formatted at 2026-09-24 10:35:17 on dist-test-slave-8wc7"
server_key: "50eefed958fc6861f85a2186b29aadf9"
server_key_iv: "19d0ed696227292aceaa457027d697e9"
server_key_version: "encryptionkey@0"
I20260924 10:35:17.781427 8928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 10:35:17.783864 8943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:17.785104 8928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:17.785279 8928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "e3f037e6aa20415a9226a44178f67e35"
format_stamp: "Formatted at 2026-09-24 10:35:17 on dist-test-slave-8wc7"
server_key: "50eefed958fc6861f85a2186b29aadf9"
server_key_iv: "19d0ed696227292aceaa457027d697e9"
server_key_version: "encryptionkey@0"
I20260924 10:35:17.785410 8928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:17.799227 8928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:17.802203 8928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:17.802421 8928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:17.809603 8928 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:40919
I20260924 10:35:17.809882 8995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:40919 every 8 connection(s)
I20260924 10:35:17.810634 8928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:17.813421 8996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:17.818746 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 8928
I20260924 10:35:17.818910 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:17.819106 8996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: Bootstrap starting.
I20260924 10:35:17.819236 2822 external_mini_cluster.cc:1428] Setting key 7ac4d4f372d6424bd2700bac98b087d3
I20260924 10:35:17.821730 8996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:17.822494 8996 log.cc:824] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:17.824397 8996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: No bootstrap required, opened a new log
Sep 24 10:35:17 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:17.829068 8996 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } }
I20260924 10:35:17.829376 8996 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:17.829450 8996 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f037e6aa20415a9226a44178f67e35, State: Initialized, Role: FOLLOWER
I20260924 10:35:17.829910 8996 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } }
I20260924 10:35:17.830060 8996 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:17.830142 8996 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:17.830241 8996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:17.831187 8996 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } }
I20260924 10:35:17.831542 8996 leader_election.cc:302] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f037e6aa20415a9226a44178f67e35; no voters:
I20260924 10:35:17.831902 8996 leader_election.cc:288] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:17.832021 9001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:17.832257 9001 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 1 LEADER]: Becoming Leader. State: Replica: e3f037e6aa20415a9226a44178f67e35, State: Running, Role: LEADER
I20260924 10:35:17.832696 9001 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } }
I20260924 10:35:17.833359 8996 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:17.834448 9003 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f037e6aa20415a9226a44178f67e35. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } }
I20260924 10:35:17.834568 9003 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:17.835058 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:17.820668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.2.193.190:40919)
0924 10:35:17.821113 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:17.821129 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:17.821165 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:35:17.821932 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:17.821950 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:17.822291 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:17.822645 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:17.823476 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:17.824452 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:17.825643 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:17.826117 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:17.826240 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:17.829265 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:17.829272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:17.829290 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:17.829330 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:17.831252 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:17.831754 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:17.831761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:17.831767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:17.831919 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:17.832303 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:17.832307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:17.832309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:17.832823 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:17.833035 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:17.833459 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 10:35:17.833786 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
I20260924 10:35:17.835032 9002 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } }
I20260924 10:35:17.835162 9002 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
W20260924 10:35:17.838948 9017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:35:17.839022 9017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:35:17.839128 9010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:17.839811 9010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:17.845731 9010 catalog_manager.cc:1347] Generated new cluster ID: 1a1fb670992b47c594a3927d6bcc75f5
I20260924 10:35:17.845813 9010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:17.869088 9010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:17.870448 9010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:17.875280 9010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: Generated new TSK 0
I20260924 10:35:17.876139 9010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab.
Entry for principal oryx/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab.
I20260924 10:35:17.912328 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40919
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab
--principal=oryx/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:18.031044 9022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:18.031334 9022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:18.031379 9022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:18.035843 9022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:18.035923 9022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:18.036008 9022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:18.041563 9022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab
--principal=oryx/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:18.043012 9022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:18.044174 9022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:18.054059 9027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.054046 9028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:18.054168 9022 server_base.cc:1034] running on GCE node
W20260924 10:35:18.054617 9030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:18.055042 9022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:18.055703 9022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:18.056908 9022 hybrid_clock.cc:648] HybridClock initialized: now 1790246118056877 us; error 50 us; skew 500 ppm
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:18.059752 9022 init.cc:377] Logged in from keytab as oryx/127.2.193.129@KRBTEST.COM (short username oryx)
I20260924 10:35:18.060693 9022 webserver.cc:460] Webserver started at http://127.2.193.129:46625/ using document root <none> and password file <none>
I20260924 10:35:18.061321 9022 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:18.061407 9022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:18.061641 9022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:18.063688 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "ae8bb19345504f579f0b7e49c8e4f96e"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "1b27fe57822e95216adc1f0c51855222"
server_key_iv: "0addcb0a1dfb8529b16b42c9fdedcf65"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.064247 9022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "ae8bb19345504f579f0b7e49c8e4f96e"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "1b27fe57822e95216adc1f0c51855222"
server_key_iv: "0addcb0a1dfb8529b16b42c9fdedcf65"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.068044 9022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:18.070518 9037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.071794 9022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:18.071935 9022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "ae8bb19345504f579f0b7e49c8e4f96e"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "1b27fe57822e95216adc1f0c51855222"
server_key_iv: "0addcb0a1dfb8529b16b42c9fdedcf65"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.072052 9022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:18.089334 9022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:18.092525 9022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:18.092736 9022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:18.093401 9022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:18.094239 9022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:18.094306 9022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.094370 9022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:18.094410 9022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.103088 9022 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38325
I20260924 10:35:18.103111 9150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38325 every 8 connection(s)
I20260924 10:35:18.104003 9022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:18.109309 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 9022
I20260924 10:35:18.109441 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:18.109730 2822 external_mini_cluster.cc:1428] Setting key 310dd47da804bf0b40f635267baf7808
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:35:18.118950 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.105840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:48323 (local address 127.2.193.190:40919)
0924 10:35:18.106005 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.106010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.106930 (+ 920us) server_negotiation.cc:407] Connection header received
0924 10:35:18.108215 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.108217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.108267 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.108346 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.109700 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.110406 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.111981 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.112212 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.112290 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.115218 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.115222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.115227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.115262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.116832 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.117420 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.117424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.117425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.117506 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.117839 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.117842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.117844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.118036 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.118140 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.118652 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.118781 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:35:18.119796 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.106268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.129:48323)
0924 10:35:18.106763 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.106808 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.108020 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.108518 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.108525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.108903 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.109547 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.109570 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.110530 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.110533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.111836 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.111851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.112374 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.112380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.112616 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.113369 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.113385 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.115021 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.116960 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.116966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.116983 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.117321 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.117597 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.117601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.117603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.117743 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.118136 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.118142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.118405 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.118601 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.118942 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":156,"threads_started":1}
I20260924 10:35:18.121066 9151 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40919
I20260924 10:35:18.121394 9151 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:18.122000 9151 heartbeater.cc:511] Master 127.2.193.190:40919 requested a full tablet report, sending...
I20260924 10:35:18.123620 8960 ts_manager.cc:194] Registered new tserver with Master: ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:18.125363 8960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:48323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab.
Entry for principal oryx/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab.
I20260924 10:35:18.138293 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40919
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab
--principal=oryx/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:18.258298 9156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:18.258675 9156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:18.258751 9156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:18.262976 9156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:18.263047 9156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:18.263110 9156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:18.268661 9156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab
--principal=oryx/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:18.269867 9156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:18.270983 9156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:18.280890 9164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.280903 9161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.280894 9162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:18.281132 9156 server_base.cc:1034] running on GCE node
I20260924 10:35:18.281641 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:18.282316 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:18.283519 9156 hybrid_clock.cc:648] HybridClock initialized: now 1790246118283503 us; error 34 us; skew 500 ppm
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:18.286437 9156 init.cc:377] Logged in from keytab as oryx/127.2.193.130@KRBTEST.COM (short username oryx)
I20260924 10:35:18.287364 9156 webserver.cc:460] Webserver started at http://127.2.193.130:40507/ using document root <none> and password file <none>
I20260924 10:35:18.287917 9156 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:18.287978 9156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:18.288177 9156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:18.290337 9156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "ba188de6cd214f1cad09f370758fb39c"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "6812c79892f917727ff9a76bc7fcc074"
server_key_iv: "2fc139c19f5b4e8ed0a5d8c0a1f806f3"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.290858 9156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "ba188de6cd214f1cad09f370758fb39c"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "6812c79892f917727ff9a76bc7fcc074"
server_key_iv: "2fc139c19f5b4e8ed0a5d8c0a1f806f3"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.294741 9156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:18.297153 9171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.298454 9156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:18.298560 9156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "ba188de6cd214f1cad09f370758fb39c"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "6812c79892f917727ff9a76bc7fcc074"
server_key_iv: "2fc139c19f5b4e8ed0a5d8c0a1f806f3"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.298653 9156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:18.315724 9156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:18.319209 9156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:18.319379 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:18.319949 9156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:18.320742 9156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:18.320789 9156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.320832 9156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:18.320853 9156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.330099 9156 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:36309
I20260924 10:35:18.330117 9284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:36309 every 8 connection(s)
I20260924 10:35:18.331073 9156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:18.335152 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 9156
I20260924 10:35:18.335297 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:18.335597 2822 external_mini_cluster.cc:1428] Setting key 4238edb2b8d33d5855d38d41edd6ea5e
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:18.346207 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.332963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:51731 (local address 127.2.193.190:40919)
0924 10:35:18.333106 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.333110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.334107 (+ 997us) server_negotiation.cc:407] Connection header received
0924 10:35:18.335854 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.335858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.335909 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.335995 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.337250 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.337778 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.339363 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.339588 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.339697 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.342510 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.342514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.342517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.342545 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.344036 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.344585 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.344588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.344590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.344676 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.345012 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.345015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.345017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.345275 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.345387 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.345908 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.346040 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:35:18.347158 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.333435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:51731)
0924 10:35:18.333959 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.334006 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.335562 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.336181 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.336188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.336576 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.337068 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.337082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.337908 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.337915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.339239 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.339248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.339735 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.339739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.339955 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.340582 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.340608 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.342334 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.344168 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.344174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.344191 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.344476 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.344764 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.344769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.344771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.344914 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.345383 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.345390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.345659 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.345864 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.346259 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.131@KRBTEST.COM" created.
I20260924 10:35:18.348451 9285 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40919
I20260924 10:35:18.348716 9285 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:18.349287 9285 heartbeater.cc:511] Master 127.2.193.190:40919 requested a full tablet report, sending...
I20260924 10:35:18.350327 8960 ts_manager.cc:194] Registered new tserver with Master: ba188de6cd214f1cad09f370758fb39c (127.2.193.130:36309)
I20260924 10:35:18.350900 8960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:51731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab.
Entry for principal oryx/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab.
I20260924 10:35:18.364430 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40919
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab
--principal=oryx/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:18.477447 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:18.477710 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:18.477792 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:18.481835 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:18.481895 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:18.481982 9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:18.487279 9290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab
--principal=oryx/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:18.488348 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:18.489470 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:18.499046 9296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.499048 9298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.499046 9295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:18.499194 9290 server_base.cc:1034] running on GCE node
I20260924 10:35:18.499737 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:18.500339 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:18.501490 9290 hybrid_clock.cc:648] HybridClock initialized: now 1790246118501475 us; error 26 us; skew 500 ppm
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:18.504048 9290 init.cc:377] Logged in from keytab as oryx/127.2.193.131@KRBTEST.COM (short username oryx)
I20260924 10:35:18.504887 9290 webserver.cc:460] Webserver started at http://127.2.193.131:33629/ using document root <none> and password file <none>
I20260924 10:35:18.505496 9290 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:18.505580 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:18.505793 9290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:18.507768 9290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "7b504c9120404b93a9f5752c140b8013"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "43b82ea2943d2dacfb92e4f2ec3b882b"
server_key_iv: "7fb5b47d15ebfc8b6c65078bf7a8be7c"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.508323 9290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "7b504c9120404b93a9f5752c140b8013"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "43b82ea2943d2dacfb92e4f2ec3b882b"
server_key_iv: "7fb5b47d15ebfc8b6c65078bf7a8be7c"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.511891 9290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:35:18.514191 9305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.515309 9290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:18.515429 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "7b504c9120404b93a9f5752c140b8013"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "43b82ea2943d2dacfb92e4f2ec3b882b"
server_key_iv: "7fb5b47d15ebfc8b6c65078bf7a8be7c"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.515534 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:18.531708 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:18.535115 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:18.535331 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:18.535939 9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:18.536737 9290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:18.536801 9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.536864 9290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:18.536909 9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.545372 9290 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:45253
I20260924 10:35:18.545396 9418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:45253 every 8 connection(s)
I20260924 10:35:18.546295 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:18.550947 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 9290
I20260924 10:35:18.551080 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:18.551412 2822 external_mini_cluster.cc:1428] Setting key 69920488be170786d1b8ced8c611a201
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:18.559773 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.548047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:34353 (local address 127.2.193.190:40919)
0924 10:35:18.548186 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.548190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.549451 (+ 1261us) server_negotiation.cc:407] Connection header received
0924 10:35:18.550327 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.550330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.550382 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.550460 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.551927 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.552535 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.553711 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.553950 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.554064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.556420 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.556423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.556426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.556450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.557848 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.558324 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.558328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.558330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.558407 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.558729 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.558734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.558735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.558918 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.559026 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.559493 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.559614 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:35:18.560639 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.548432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.131:34353)
0924 10:35:18.549317 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.549360 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.550132 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.550631 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.550638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.551132 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.551788 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.551805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.552656 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.552660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.553596 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.553605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.554095 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.554098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.554256 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.554827 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.554844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.556271 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.557956 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.557961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.557977 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.558224 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.558496 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.558500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.558502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.558640 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.559023 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.559030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.559273 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.559455 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.559786 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":538,"threads_started":1}
I20260924 10:35:18.561725 9419 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40919
I20260924 10:35:18.561954 9419 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:18.562424 9419 heartbeater.cc:511] Master 127.2.193.190:40919 requested a full tablet report, sending...
I20260924 10:35:18.563402 8960 ts_manager.cc:194] Registered new tserver with Master: 7b504c9120404b93a9f5752c140b8013 (127.2.193.131:45253)
I20260924 10:35:18.564083 8960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:34353
I20260924 10:35:18.565282 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.193.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.189.keytab.
Entry for principal oryx/127.2.193.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.189.keytab.
I20260924 10:35:18.591835 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:34413
--webserver_interface=127.2.193.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.189.keytab
--principal=oryx/127.2.193.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40919,127.2.193.189:34413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:18.707705 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:18.707999 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:18.708041 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:18.712208 9424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:18.712291 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:18.712311 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:18.712327 9424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:18.712342 9424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:18.718516 9424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40919,127.2.193.189:34413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.189.keytab
--principal=oryx/127.2.193.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.193.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
W20260924 10:35:18.718753 9424 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.193.190:40919,127.2.193.189:34413'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 10:35:18.719712 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:18.720741 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:18.729565 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.729570 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:18.729660 9424 server_base.cc:1034] running on GCE node
W20260924 10:35:18.729600 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:18.730386 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:18.731626 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:18.732796 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790246118732779 us; error 33 us; skew 500 ppm
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:18.735783 9424 init.cc:377] Logged in from keytab as oryx/127.2.193.189@KRBTEST.COM (short username oryx)
I20260924 10:35:18.736778 9424 webserver.cc:460] Webserver started at http://127.2.193.189:39741/ using document root <none> and password file <none>
I20260924 10:35:18.737452 9424 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:18.737550 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:18.737812 9424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:18.739984 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/instance:
uuid: "f7028b32e8a9475ba90d969a51eb71ed"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "77c3ee820ac0024d10a79a0422764a5e"
server_key_iv: "29f3f1dcf89019cadfd418fcbe8bcbb1"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.740600 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal/instance:
uuid: "f7028b32e8a9475ba90d969a51eb71ed"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "77c3ee820ac0024d10a79a0422764a5e"
server_key_iv: "29f3f1dcf89019cadfd418fcbe8bcbb1"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.744592 9424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:18.747246 9439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.748579 9424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:35:18.748718 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
uuid: "f7028b32e8a9475ba90d969a51eb71ed"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "77c3ee820ac0024d10a79a0422764a5e"
server_key_iv: "29f3f1dcf89019cadfd418fcbe8bcbb1"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.748828 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:18.794921 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:18.798718 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:18.798920 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:18.806355 9424 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.189:34413
I20260924 10:35:18.806641 9491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.189:34413 every 8 connection(s)
I20260924 10:35:18.807454 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
I20260924 10:35:18.808632 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 9424
I20260924 10:35:18.810774 9492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:18.814991 9492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.193.189@KRBTEST.COM
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.189@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:18.826118 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.810256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.2.193.189:34413)
0924 10:35:18.810731 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.810749 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.810792 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:35:18.811624 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.811648 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.812086 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.812461 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.813391 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.814936 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.816090 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.816809 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.817431 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.820089 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.820096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.820108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.820138 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.822764 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.823266 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.823273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.823280 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.823408 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.823732 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.823735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.823737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.824291 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.824539 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.824740 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.825040 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":166,"threads_started":1}
I20260924 10:35:18.826709 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.816921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.2.193.190:40919)
0924 10:35:18.817072 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.817077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.817455 (+ 378us) server_negotiation.cc:407] Connection header received
0924 10:35:18.817649 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.817653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.817719 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.817763 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.818607 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.819426 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.820641 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.820988 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.821096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.823303 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.823306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.823308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.823330 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.824934 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.825429 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.825432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.825434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.825518 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.825765 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.825768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.825770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.826011 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.826109 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.826452 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.826559 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:35:18.826736 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.816912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:53244)
0924 10:35:18.817332 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.817381 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.817544 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.817904 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.817908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.818361 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.818492 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.818498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.819653 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.819658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.820517 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.820529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.821103 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.821106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.821239 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.821714 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.821732 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.823161 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.825056 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.825062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.825067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.825342 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.825605 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.825607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.825608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.825678 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.826099 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.826104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.826213 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.826380 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.826560 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":131,"threads_started":1}
I20260924 10:35:18.827905 9492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.189@KRBTEST.COM for oryx/127.2.193.189@KRBTEST.COM
I20260924 10:35:18.835907 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.828451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.0.0.1:36498)
0924 10:35:18.828560 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.828574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.828644 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.828814 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.828816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.828945 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.829028 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.829032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.830075 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.830077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.830593 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.830600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.830974 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.830977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.831064 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.831532 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.831543 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.832709 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.834431 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.834437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.834439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.834625 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.834880 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.834884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.834886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.834977 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.835367 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.835370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.835451 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.835562 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.835703 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":15}
I20260924 10:35:18.835907 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.828512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.2.193.189:34413)
0924 10:35:18.828615 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.828619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.828636 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:35:18.828701 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.828703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.828733 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.828793 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.829114 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.829985 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.830683 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.830890 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.830972 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.832843 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.832846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.832848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.832866 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.834336 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.834726 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.834730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.834732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.834798 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.835062 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.835068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.835069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.835275 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.835379 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.835603 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.835752 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":41}
I20260924 10:35:18.839000 9492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Bootstrap starting.
I20260924 10:35:18.841434 9492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:18.842140 9492 log.cc:824] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:18.843986 9492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: No bootstrap required, opened a new log
I20260924 10:35:18.846971 9492 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } }
I20260924 10:35:18.847169 9492 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:18.847203 9492 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7028b32e8a9475ba90d969a51eb71ed, State: Initialized, Role: FOLLOWER
I20260924 10:35:18.847587 9492 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } }
I20260924 10:35:18.848037 9498 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } } }
I20260924 10:35:18.848166 9498 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:35:18.848770 9492 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:35:18.853471 9509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:35:18.853595 9509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:35:18.855427 9424 master_runner.cc:428] Detected that this master f7028b32e8a9475ba90d969a51eb71ed is joining an existing cluster
I20260924 10:35:18.855480 9424 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.193.189:34413
I20260924 10:35:18.866091 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.857905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.2.193.189:34413)
0924 10:35:18.858006 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.858012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.858026 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:18.858142 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.858145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.858184 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.858253 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.858900 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.859655 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.860817 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.861020 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.861128 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.862867 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.862871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.862873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.862897 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.864262 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.864744 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.864748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.864750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.864829 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.865085 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.865092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.865094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.865321 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.865428 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.865768 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.865895 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":40}
I20260924 10:35:18.866636 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.857585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.0.0.1:36514)
0924 10:35:18.857956 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.857968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.858057 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.858287 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.858290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.858611 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.858744 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.858752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.859778 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.859781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.860707 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.860717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.861135 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.861138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.861262 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.862248 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.862263 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.862729 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.864364 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.864366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.864368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.864630 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.864903 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.864906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.864908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.864996 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.865475 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.865479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.865589 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.866302 (+ 713us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.866436 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":7,"thread_start_us":93,"threads_started":1}
I20260924 10:35:18.868611 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.857509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:53250)
0924 10:35:18.857773 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.857789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.857953 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.858341 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.858344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.858590 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.858699 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.858705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.860857 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.860860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.861775 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.861786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.862451 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.862455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.862549 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.863182 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.863195 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.864746 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.867064 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.867067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.867070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.867325 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.867652 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.867654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.867656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.867732 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.868109 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.868112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.868195 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.868299 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.868434 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":113,"threads_started":1}
I20260924 10:35:18.868613 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.857563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.2.193.190:40919)
0924 10:35:18.857708 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.857715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.857878 (+ 163us) server_negotiation.cc:407] Connection header received
0924 10:35:18.858088 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.858091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.858144 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.858218 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.858812 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.859655 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.861931 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.862328 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.862443 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.864894 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.864898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.864904 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.864932 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.866599 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.867434 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.867437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.867439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.867541 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.867822 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.867827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.867829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.868017 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.868122 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.868354 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.868472 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246117, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.190@KRBTEST.COM for oryx/127.2.193.189@KRBTEST.COM
I20260924 10:35:18.882743 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.872768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.2.193.189:34413)
0924 10:35:18.872894 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.872899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.873218 (+ 319us) server_negotiation.cc:407] Connection header received
0924 10:35:18.873420 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.873423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.873475 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.873558 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.874172 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.874941 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.875943 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.876284 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.876389 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.878667 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.878671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.878674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.878700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.880589 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.881211 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.881214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.881216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.881312 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.881643 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.881647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.881648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.881933 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.882032 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.882456 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.882580 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:35:18.882953 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.872721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.0.0.1:36520)
0924 10:35:18.873076 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.873128 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.873286 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.873599 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.873604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.873962 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.874067 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.874073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.875038 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.875040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.875841 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.875850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.876385 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.876387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.876499 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.877057 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.877068 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.878499 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.880708 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.880722 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.880733 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.881080 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.881413 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.881423 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.881426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.881548 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.882049 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.882059 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.882218 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.882393 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.882649 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260924 10:35:18.884372 8960 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.193.189:34413: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed"
member_type: NON_VOTER
last_known_addr {
host: "127.2.193.189"
port: 34413
}
attrs {
promote: true
}
}
dest_uuid: "e3f037e6aa20415a9226a44178f67e35"
cas_config_opid_index: -1
I20260924 10:35:18.885025 8960 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } }
I20260924 10:35:18.886940 9521 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f7028b32e8a9475ba90d969a51eb71ed (127.2.193.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } }
I20260924 10:35:18.887625 9466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:18.888923 9466 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Refusing update from remote peer e3f037e6aa20415a9226a44178f67e35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 10:35:18.889010 9521 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.193.189:34413
I20260924 10:35:18.889467 9520 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } }
I20260924 10:35:18.889580 9520 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:18.889624 9424 master.cc:537] Master@127.2.193.189:34413 shutting down...
I20260924 10:35:18.889767 9520 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.2.193.188@KRBTEST.COM; defaulting to no policy
W20260924 10:35:18.892580 8944 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 10:35:18.893906 8944 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 -> Peer f7028b32e8a9475ba90d969a51eb71ed (127.2.193.189:34413): Couldn't send request to peer f7028b32e8a9475ba90d969a51eb71ed. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.188@KRBTEST.COM" created.
I20260924 10:35:18.894588 9424 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:18.894910 9424 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:18.895040 9424 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: stopping tablet replica
I20260924 10:35:18.899827 9424 master.cc:552] Master@127.2.193.189:34413 shutdown complete.
I20260924 10:35:18.899889 9424 master_runner.cc:305] Clearing existing system tablet
I20260924 10:35:18.900175 9424 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:18.900247 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:18.901976 9424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 10:35:18.903000 9526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.903256 9424 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:18.903586 9424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 10:35:18.903685 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
uuid: "f7028b32e8a9475ba90d969a51eb71ed"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "77c3ee820ac0024d10a79a0422764a5e"
server_key_iv: "29f3f1dcf89019cadfd418fcbe8bcbb1"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.906052 9424 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:18.908655 9424 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:35:18.908768 9424 log.cc:1196] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 10:35:18.909051 9424 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Deleting consensus metadata
I20260924 10:35:18.909267 9424 master_runner.cc:315] Copying system tablet from 127.2.193.190:40919
I20260924 10:35:18.910457 9424 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: tablet copy: Beginning tablet copy session from remote peer at address 127.2.193.190:40919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.188.keytab.
Entry for principal oryx/127.2.193.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.188.keytab.
I20260924 10:35:18.913447 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:45491
--webserver_interface=127.2.193.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.188.keytab
--principal=oryx/127.2.193.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:35:18.919516 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.911154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.2.193.190:40919)
0924 10:35:18.911292 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:18.911298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:18.911416 (+ 118us) server_negotiation.cc:407] Connection header received
0924 10:35:18.911552 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:18.911554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:18.911622 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:18.911704 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:18.912211 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.912957 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.914131 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.914352 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.914469 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.915972 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:18.915975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:18.915979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:18.916009 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:18.917616 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.918121 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.918126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.918128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.918214 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:18.918570 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:18.918573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:18.918575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:18.918799 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:18.918903 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:18.919237 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 10:35:18.919355 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:35:18.920017 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:18.911021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:53260)
0924 10:35:18.911309 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:18.911325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:18.911458 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:18.911727 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:18.911730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:18.911953 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:18.912086 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.912093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.913075 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.913080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.913997 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:18.914008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:18.914464 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:18.914468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:18.914563 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:18.915298 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:18.915317 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:18.915805 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:18.917726 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.917732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.917737 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.918015 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.918310 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:18.918317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:18.918321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:18.918424 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:18.918907 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:18.918912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:18.919044 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 10:35:18.919176 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 10:35:18.919358 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260924 10:35:18.920598 8980 tablet_copy_service.cc:140] P e3f037e6aa20415a9226a44178f67e35: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f7028b32e8a9475ba90d969a51eb71ed ({username='oryx', principal='oryx/127.2.193.189@KRBTEST.COM'} at 127.0.0.1:53260)
I20260924 10:35:18.920792 8980 tablet_copy_service.cc:161] P e3f037e6aa20415a9226a44178f67e35: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f7028b32e8a9475ba90d969a51eb71ed at {username='oryx', principal='oryx/127.2.193.189@KRBTEST.COM'} at 127.0.0.1:53260: session id = f7028b32e8a9475ba90d969a51eb71ed-00000000000000000000000000000000
I20260924 10:35:18.922669 8980 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:35:18.924544 9424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:35:18.927340 9424 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: tablet copy: Starting download of 0 data blocks...
I20260924 10:35:18.927541 9424 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: tablet copy: Starting download of 1 WAL segments...
I20260924 10:35:18.929425 9424 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:35:18.931300 8980 tablet_copy_service.cc:342] P e3f037e6aa20415a9226a44178f67e35: Request end of tablet copy session f7028b32e8a9475ba90d969a51eb71ed-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.193.189@KRBTEST.COM'} at 127.0.0.1:53260
I20260924 10:35:18.931432 8980 tablet_copy_service.cc:433] P e3f037e6aa20415a9226a44178f67e35: ending tablet copy session f7028b32e8a9475ba90d969a51eb71ed-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f7028b32e8a9475ba90d969a51eb71ed
I20260924 10:35:18.940733 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:18.942696 9534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.942700 9535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:18.945362 9537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:18.945528 9424 server_base.cc:1034] running on GCE node
I20260924 10:35:18.945789 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:18.946137 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:18.947289 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790246118947271 us; error 42 us; skew 500 ppm
Sep 24 10:35:18 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246118, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:18.948948 9424 init.cc:377] Logged in from keytab as oryx/127.2.193.189@KRBTEST.COM (short username oryx)
I20260924 10:35:18.949568 9424 webserver.cc:460] Webserver started at http://127.2.193.189:42083/ using document root <none> and password file <none>
I20260924 10:35:18.949762 9424 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:18.949805 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:18.951140 9424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:18.952157 9544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:18.952392 9424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:18.952472 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
uuid: "f7028b32e8a9475ba90d969a51eb71ed"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "77c3ee820ac0024d10a79a0422764a5e"
server_key_iv: "29f3f1dcf89019cadfd418fcbe8bcbb1"
server_key_version: "encryptionkey@0"
I20260924 10:35:18.952548 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:18.982148 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:18.984596 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:18.989485 9424 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.189:34413
I20260924 10:35:18.989509 9596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.189:34413 every 8 connection(s)
I20260924 10:35:18.990378 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
I20260924 10:35:18.992391 9597 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:35:18.993510 9597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Bootstrap starting.
I20260924 10:35:19.004446 9597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:35:19.004822 9597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: Bootstrap complete.
I20260924 10:35:19.005503 9597 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } }
I20260924 10:35:19.005653 9597 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7028b32e8a9475ba90d969a51eb71ed, State: Initialized, Role: LEARNER
I20260924 10:35:19.005782 9597 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } }
I20260924 10:35:19.006166 9600 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } }
I20260924 10:35:19.006286 9597 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:19.006387 9600 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: LEARNER
I20260924 10:35:19.009269 9611 catalog_manager.cc:1259] Loaded cluster ID: 1a1fb670992b47c594a3927d6bcc75f5
I20260924 10:35:19.009433 9611 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: loading cluster ID for follower catalog manager: success
I20260924 10:35:19.011766 9611 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: acquiring CA information for follower catalog manager: success
I20260924 10:35:19.013298 9611 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:35:19.054342 9533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:19.054667 9533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:19.054754 9533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:19.059019 9533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:19.059121 9533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:19.059160 9533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:19.059196 9533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:19.059247 9533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:19.064946 9533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.188.keytab
--principal=oryx/127.2.193.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:45491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.193.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:19.066186 9533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:19.067340 9533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:19.076076 9617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.076078 9619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.076078 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:19.076520 9533 server_base.cc:1034] running on GCE node
I20260924 10:35:19.077229 9533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:19.078418 9533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:19.079582 9533 hybrid_clock.cc:648] HybridClock initialized: now 1790246119079547 us; error 55 us; skew 500 ppm
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:19.082523 9533 init.cc:377] Logged in from keytab as oryx/127.2.193.188@KRBTEST.COM (short username oryx)
I20260924 10:35:19.083482 9533 webserver.cc:460] Webserver started at http://127.2.193.188:41981/ using document root <none> and password file <none>
I20260924 10:35:19.084070 9533 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:19.084133 9533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:19.084373 9533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:19.086568 9533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/instance:
uuid: "41eb1f74cba6410d98bd5768c7626cfb"
format_stamp: "Formatted at 2026-09-24 10:35:19 on dist-test-slave-8wc7"
server_key: "6212661832157c7f679b92ad739f7126"
server_key_iv: "8092556bb759cf421deb494511628255"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.087162 9533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal/instance:
uuid: "41eb1f74cba6410d98bd5768c7626cfb"
format_stamp: "Formatted at 2026-09-24 10:35:19 on dist-test-slave-8wc7"
server_key: "6212661832157c7f679b92ad739f7126"
server_key_iv: "8092556bb759cf421deb494511628255"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.091269 9533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:19.093856 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.095136 9533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:19.095278 9533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
uuid: "41eb1f74cba6410d98bd5768c7626cfb"
format_stamp: "Formatted at 2026-09-24 10:35:19 on dist-test-slave-8wc7"
server_key: "6212661832157c7f679b92ad739f7126"
server_key_iv: "8092556bb759cf421deb494511628255"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.095395 9533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:19.127972 9151 heartbeater.cc:503] Master 127.2.193.190:40919 was elected leader, sending a full tablet report...
I20260924 10:35:19.133912 9533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:19.137619 9533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:19.137848 9533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:19.145412 9533 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.188:45491
I20260924 10:35:19.145691 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.188:45491 every 8 connection(s)
I20260924 10:35:19.146469 9533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
I20260924 10:35:19.149318 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:19.152532 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 9533
I20260924 10:35:19.153318 9679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.188@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.193.188@KRBTEST.COM
I20260924 10:35:19.165679 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.155311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.2.193.190:40919)
0924 10:35:19.155436 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.155440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.155625 (+ 185us) server_negotiation.cc:407] Connection header received
0924 10:35:19.155986 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.155989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.156040 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.156134 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.157212 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.157728 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.158852 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.159061 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.159141 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.161900 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.161903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.161905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.161929 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.163511 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.164111 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.164116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.164121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.164212 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.164549 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.164554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.164556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.164744 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.164846 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.165380 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.165507 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:35:19.166594 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.155097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:53270)
0924 10:35:19.155480 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.155530 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.155783 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.156195 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.156200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.156707 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.157037 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.157049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.157852 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.157856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.158698 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.158707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.159198 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.159202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.159446 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.160113 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.160129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.161682 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.163632 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.163639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.163647 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.164011 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.164309 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.164312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.164314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.164456 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.164844 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.164851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.165111 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.165337 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.165658 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":53,"thread_start_us":131,"threads_started":1}
I20260924 10:35:19.166599 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.154183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.2.193.188:45491)
0924 10:35:19.154595 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.154609 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.154643 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:35:19.155480 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.155502 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.155791 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.156036 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.156959 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.158013 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.158806 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.159220 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.159416 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.162235 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.162240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.162250 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.162270 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.164038 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.164573 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.164578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.164580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.164660 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.164943 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.164946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.164948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.165263 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.165501 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.165710 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.165850 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260924 10:35:19.168525 9679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.188@KRBTEST.COM for oryx/127.2.193.189@KRBTEST.COM
I20260924 10:35:19.176522 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.168992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.0.0.1:36524)
0924 10:35:19.169103 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.169117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.169227 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.169566 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.169568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.169698 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.169788 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.169792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.170631 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.170633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.171170 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.171176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.171554 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.171556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.171625 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.172044 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.172055 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.173299 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.175019 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.175021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.175023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.175406 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.175671 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.175673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.175674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.175747 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.176066 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.176068 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.176147 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.176224 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.176395 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 10:35:19.176741 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.169102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.2.193.189:34413)
0924 10:35:19.169381 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.169385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.169396 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:35:19.169431 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.169434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.169471 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.169546 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.169884 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.170545 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.171256 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.171471 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.171545 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.173625 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.173627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.173630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.173654 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.174909 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.175515 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.175518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.175519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.175594 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.175821 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.175823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.175824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.175993 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.176067 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.176264 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.176533 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":138,"threads_started":1}
I20260924 10:35:19.177472 9679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.188@KRBTEST.COM for oryx/127.2.193.188@KRBTEST.COM
I20260924 10:35:19.184535 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.177972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.0.0.1:36286)
0924 10:35:19.178094 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.178108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.178174 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.178360 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.178362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.178487 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.178559 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.178563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.179248 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.179250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.179743 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.179749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.180138 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.180140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.180217 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.180631 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.180642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.181820 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.183371 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.183374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.183375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.183546 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.183754 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.183756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.183757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.183824 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.184129 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.184131 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.184199 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.184274 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.184386 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":14}
I20260924 10:35:19.184574 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.178038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.2.193.188:45491)
0924 10:35:19.178145 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.178148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.178159 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:35:19.178242 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.178244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.178276 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.178337 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.178641 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.179154 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.179829 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.180054 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.180127 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.181944 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.181947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.181949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.181974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.183286 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.183619 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.183621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.183622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.183685 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.183897 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.183899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.183900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.184060 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.184130 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.184320 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.184436 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":42}
I20260924 10:35:19.187979 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Bootstrap starting.
I20260924 10:35:19.190478 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:19.191219 9679 log.cc:824] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:19.193221 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: No bootstrap required, opened a new log
I20260924 10:35:19.196221 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } }
I20260924 10:35:19.196460 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:19.196537 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41eb1f74cba6410d98bd5768c7626cfb, State: Initialized, Role: FOLLOWER
I20260924 10:35:19.196974 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } }
I20260924 10:35:19.197474 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } } }
I20260924 10:35:19.197587 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:35:19.198221 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:19.199483 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:19.199702 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } }
I20260924 10:35:19.201339 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41eb1f74cba6410d98bd5768c7626cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f037e6aa20415a9226a44178f67e35" is_pre_election: true
I20260924 10:35:19.201534 9571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41eb1f74cba6410d98bd5768c7626cfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7028b32e8a9475ba90d969a51eb71ed" is_pre_election: true
I20260924 10:35:19.201705 9571 raft_consensus.cc:1799] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 LEARNER]: Handling vote request from an unknown peer 41eb1f74cba6410d98bd5768c7626cfb
I20260924 10:35:19.201853 9571 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 41eb1f74cba6410d98bd5768c7626cfb for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 10:35:19.200981 9685 leader_election.cc:288] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919), f7028b32e8a9475ba90d969a51eb71ed (127.2.193.189:34413)
I20260924 10:35:19.202222 9630 leader_election.cc:302] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41eb1f74cba6410d98bd5768c7626cfb; no voters: e3f037e6aa20415a9226a44178f67e35, f7028b32e8a9475ba90d969a51eb71ed
I20260924 10:35:19.202471 9685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:19.203928 9685 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 10:35:19.204447 9696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:35:19.204526 9696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:35:19.207165 9533 master_runner.cc:428] Detected that this master 41eb1f74cba6410d98bd5768c7626cfb is joining an existing cluster
I20260924 10:35:19.207240 9533 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.193.188:45491
I20260924 10:35:19.220037 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 9022
I20260924 10:35:19.220512 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.210862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.2.193.190:40919)
0924 10:35:19.211012 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.211017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.212143 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 10:35:19.212208 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.212211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.212263 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.212346 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.212776 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.213595 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.214873 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.215210 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.215323 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.216613 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.216618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.216621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.216653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.218301 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.218888 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.218892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.218894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.218998 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.219329 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.219332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.219334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.219604 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.219711 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.220126 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.220277 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:35:19.220520 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.211105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:53286)
0924 10:35:19.212021 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.212035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.212137 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.212359 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.212362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.212553 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.212652 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.212659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.213710 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.213716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.214743 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.214752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.215355 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.215358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.215442 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.216046 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.216057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.216455 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.218521 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.218525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.218527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.218751 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.219097 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.219101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.219102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.219214 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.219802 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.219806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.219923 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.220038 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.220322 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":90,"threads_started":1}
I20260924 10:35:19.227442 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:38325
--local_ip_for_outbound_sockets=127.2.193.129
--tserver_master_addrs=127.2.193.190:40919
--webserver_port=46625
--webserver_interface=127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab
--principal=oryx/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:35:19.234293 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.209701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.0.0.1:36528)
0924 10:35:19.210631 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.210649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.210862 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.211544 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.211575 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.212423 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.212579 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.212587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.213484 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.213487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.214892 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.214899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.215492 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.215495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.215598 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.229669 (+ 14071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.229684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.230137 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.232379 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.232384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.232389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.232667 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.233010 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.233016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.233021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.233116 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.233689 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.233695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.233809 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.233929 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.234093 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":133,"threads_started":1}
I20260924 10:35:19.234930 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.209755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.2.193.189:34413)
0924 10:35:19.209899 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.209904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.210748 (+ 844us) server_negotiation.cc:407] Connection header received
0924 10:35:19.211152 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.211156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.211245 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.211339 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.212726 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.213374 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.215001 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.215248 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.215374 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.230458 (+ 15084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.230462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.230465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.230497 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.232277 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.232802 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.232809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.232814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.232905 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.233264 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.233268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.233269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.233519 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.233621 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.234572 (+ 951us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.234734 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246117, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.190@KRBTEST.COM for oryx/127.2.193.188@KRBTEST.COM
I20260924 10:35:19.238961 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.210319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.2.193.188:45491)
0924 10:35:19.210422 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.210426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.211474 (+ 1048us) server_negotiation.cc:407] Connection header received
0924 10:35:19.211618 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.211621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.211654 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.211702 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.230399 (+ 18697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.231082 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.232303 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.232649 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.232756 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.234028 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.234031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.234034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.234053 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.236886 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.237479 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.237484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.237487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.237570 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.237903 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.237908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.237910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.238152 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.238247 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.238589 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.238716 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:35:19.239526 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.210181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.0.0.1:36298)
0924 10:35:19.211398 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.211411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.211527 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.211825 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.211827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.229505 (+ 17678us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.230287 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.230294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.231209 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.231212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.232168 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.232181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.232866 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.232869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.232983 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.233477 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.233492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.233888 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.237001 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.237008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.237010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.237365 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.237680 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.237685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.237691 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.237805 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.238302 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.238308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.238410 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.239114 (+ 704us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.239274 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":98,"threads_started":1}
I20260924 10:35:19.243348 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.226620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.0.0.1:36310)
0924 10:35:19.226878 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.226892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.226978 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.227755 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.227761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.228014 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.228162 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.228173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.229400 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.229403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.230293 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.230304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.234145 (+ 3841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.234149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.234254 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.234766 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.234963 (+ 197us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.237501 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.241253 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.241257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.241260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.241526 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.242314 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.242318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.242320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.242415 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.242819 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.242822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.242919 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.243022 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.243179 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260924 10:35:19.244244 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.227112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.2.193.188:45491)
0924 10:35:19.227429 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.227438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.227452 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:19.227536 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.227539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.227607 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.228298 (+ 691us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.228440 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.229255 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.230820 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.234010 (+ 3190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.237043 (+ 3033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.237657 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.237661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.237664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.237687 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.241093 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.241646 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.241649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.241651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.242198 (+ 547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.242520 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.242523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.242525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.242718 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.243729 (+ 1011us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.243850 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.243997 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20260924 10:35:19.245121 8960 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.193.188:45491: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb"
member_type: NON_VOTER
last_known_addr {
host: "127.2.193.188"
port: 45491
}
attrs {
promote: true
}
}
dest_uuid: "e3f037e6aa20415a9226a44178f67e35"
cas_config_opid_index: 5
I20260924 10:35:19.245853 8960 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } }
I20260924 10:35:19.247231 9520 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.248793 9520 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.193.188:45491
I20260924 10:35:19.248970 9707 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.249161 9707 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:19.249548 9533 master.cc:537] Master@127.2.193.188:45491 shutting down...
W20260924 10:35:19.250649 8946 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 -> Peer 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188:45491): Couldn't send request to peer 41eb1f74cba6410d98bd5768c7626cfb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 10:35:19.255074 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.247469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.0.0.1:36534)
0924 10:35:19.250199 (+ 2730us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.250213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.250303 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.250839 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.250842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.250863 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:19.250955 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.250961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.252067 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.252070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.253412 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.253425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.254596 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.254599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.254698 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.254755 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.254852 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.254936 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2661}
I20260924 10:35:19.255105 9533 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:19.255401 9533 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:19.255497 9533 tablet_replica.cc:330] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: stopping tablet replica
I20260924 10:35:19.255548 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.249076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.2.193.189:34413)
0924 10:35:19.249295 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.249300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.250396 (+ 1096us) server_negotiation.cc:407] Connection header received
0924 10:35:19.250456 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.250458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.250525 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.250619 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:19.251081 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.251927 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.253555 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.254242 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.254350 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.254531 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.255355 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.255435 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260924 10:35:19.256278 9571 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 LEARNER]: Refusing update from remote peer e3f037e6aa20415a9226a44178f67e35: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:35:19.256731 9707 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 10:35:19.258755 9571 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.261236 9600 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f037e6aa20415a9226a44178f67e35. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.261354 9600 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: LEARNER
I20260924 10:35:19.261439 9600 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: NON_VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: true } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.261500 9600 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: LEARNER
I20260924 10:35:19.262387 9521 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: attempting to promote NON_VOTER f7028b32e8a9475ba90d969a51eb71ed to VOTER
I20260924 10:35:19.262827 9521 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } }
I20260924 10:35:19.264272 9571 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 LEARNER]: Refusing update from remote peer e3f037e6aa20415a9226a44178f67e35: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 10:35:19.264397 9533 master.cc:552] Master@127.2.193.188:45491 shutdown complete.
I20260924 10:35:19.264472 9533 master_runner.cc:305] Clearing existing system tablet
I20260924 10:35:19.264853 9520 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 10:35:19.264946 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.264664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.0.0.1:36322)
0924 10:35:19.264792 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.264865 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 10:35:19.265787 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.265552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.0.0.1:36334)
0924 10:35:19.265673 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.265717 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 10:35:19.265990 8946 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 -> Peer 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188:45491): Couldn't send request to peer 41eb1f74cba6410d98bd5768c7626cfb. Status: Network error: Client connection negotiation failed: client connection to 127.2.193.188:45491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:35:19.266189 9533 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:19.266281 9533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:19.266431 9707 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7028b32e8a9475ba90d969a51eb71ed (127.2.193.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.267896 9533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:19.267827 9571 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7028b32e8a9475ba90d969a51eb71ed (127.2.193.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.269472 9521 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.269583 9521 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:19.269837 9521 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.269948 9521 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:19.269925 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.270211 9533 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:19.270224 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.270319 9533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:19.270330 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:35:19.270351 9533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
uuid: "41eb1f74cba6410d98bd5768c7626cfb"
format_stamp: "Formatted at 2026-09-24 10:35:19 on dist-test-slave-8wc7"
server_key: "6212661832157c7f679b92ad739f7126"
server_key_iv: "8092556bb759cf421deb494511628255"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.272356 9533 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:19.274695 9533 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:35:19.274787 9533 log.cc:1196] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 10:35:19.275038 9533 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Deleting consensus metadata
I20260924 10:35:19.275203 9533 master_runner.cc:315] Copying system tablet from 127.2.193.190:40919
I20260924 10:35:19.276472 9533 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: tablet copy: Beginning tablet copy session from remote peer at address 127.2.193.190:40919
I20260924 10:35:19.286055 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.277401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:53292)
0924 10:35:19.277665 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.277683 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.277821 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.278114 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.278117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.278345 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.278475 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.278482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.279611 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.279614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.280491 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.280502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.281094 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.281097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.281215 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.281858 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.281873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.282322 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.284331 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.284335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.284337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.284605 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.284904 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.284909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.284913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.285008 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.285466 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.285470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.285575 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.285682 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.285850 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260924 10:35:19.286067 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.277110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.2.193.190:40919)
0924 10:35:19.277305 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.277310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.277786 (+ 476us) server_negotiation.cc:407] Connection header received
0924 10:35:19.277923 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.277926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.277989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.278078 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.278661 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.279485 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.280616 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.280955 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.281073 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.282478 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.282482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.282485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.282515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.284215 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.284718 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.284722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.284724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.284814 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.285100 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.285103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.285105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.285356 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.285449 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.285748 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.285871 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:35:19.286899 8980 tablet_copy_service.cc:140] P e3f037e6aa20415a9226a44178f67e35: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 41eb1f74cba6410d98bd5768c7626cfb ({username='oryx', principal='oryx/127.2.193.188@KRBTEST.COM'} at 127.0.0.1:53292)
I20260924 10:35:19.287009 8980 tablet_copy_service.cc:161] P e3f037e6aa20415a9226a44178f67e35: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 41eb1f74cba6410d98bd5768c7626cfb at {username='oryx', principal='oryx/127.2.193.188@KRBTEST.COM'} at 127.0.0.1:53292: session id = 41eb1f74cba6410d98bd5768c7626cfb-00000000000000000000000000000000
I20260924 10:35:19.288129 8980 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:35:19.289634 9533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:35:19.292137 9533 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: tablet copy: Starting download of 0 data blocks...
I20260924 10:35:19.292389 9533 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: tablet copy: Starting download of 1 WAL segments...
I20260924 10:35:19.294201 9533 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:35:19.296119 8980 tablet_copy_service.cc:342] P e3f037e6aa20415a9226a44178f67e35: Request end of tablet copy session 41eb1f74cba6410d98bd5768c7626cfb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.193.188@KRBTEST.COM'} at 127.0.0.1:53292
I20260924 10:35:19.296205 8980 tablet_copy_service.cc:433] P e3f037e6aa20415a9226a44178f67e35: ending tablet copy session 41eb1f74cba6410d98bd5768c7626cfb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 41eb1f74cba6410d98bd5768c7626cfb
I20260924 10:35:19.303627 9533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:19.305440 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.306252 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.307106 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:19.310102 9533 server_base.cc:1034] running on GCE node
I20260924 10:35:19.310289 9533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:19.310614 9533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:19.311770 9533 hybrid_clock.cc:648] HybridClock initialized: now 1790246119311752 us; error 30 us; skew 500 ppm
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:19.313462 9533 init.cc:377] Logged in from keytab as oryx/127.2.193.188@KRBTEST.COM (short username oryx)
I20260924 10:35:19.313971 9533 webserver.cc:460] Webserver started at http://127.2.193.188:43299/ using document root <none> and password file <none>
I20260924 10:35:19.314168 9533 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:19.314213 9533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:19.315577 9533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:19.316622 9728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.316881 9533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:19.316960 9533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
uuid: "41eb1f74cba6410d98bd5768c7626cfb"
format_stamp: "Formatted at 2026-09-24 10:35:19 on dist-test-slave-8wc7"
server_key: "6212661832157c7f679b92ad739f7126"
server_key_iv: "8092556bb759cf421deb494511628255"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.317045 9533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:19.333541 9533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:19.335815 9533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:19.341001 9533 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.188:45491
I20260924 10:35:19.341010 9780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.188:45491 every 8 connection(s)
I20260924 10:35:19.341596 9533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
I20260924 10:35:19.343140 9781 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:35:19.344017 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Bootstrap starting.
I20260924 10:35:19.353278 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:35:19.353612 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Bootstrap complete.
I20260924 10:35:19.353598 9285 heartbeater.cc:503] Master 127.2.193.190:40919 was elected leader, sending a full tablet report...
I20260924 10:35:19.354231 9781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } }
I20260924 10:35:19.354384 9781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41eb1f74cba6410d98bd5768c7626cfb, State: Initialized, Role: LEARNER
I20260924 10:35:19.354491 9781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } }
I20260924 10:35:19.354959 9781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:19.355392 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.355588 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: This master's current role is: LEARNER
I20260924 10:35:19.358646 9795 catalog_manager.cc:1259] Loaded cluster ID: 1a1fb670992b47c594a3927d6bcc75f5
I20260924 10:35:19.358721 9795 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: loading cluster ID for follower catalog manager: success
I20260924 10:35:19.360325 9795 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: acquiring CA information for follower catalog manager: success
I20260924 10:35:19.361505 9795 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:35:19.393028 9706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:19.393458 9706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:19.393524 9706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:19.399858 9706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:19.399976 9706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:19.400110 9706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:19.407120 9706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab
--principal=oryx/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:38325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=46625
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:19.408370 9706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:19.409403 9706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:19.419024 9801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.419047 9800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.419040 9803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:19.419294 9706 server_base.cc:1034] running on GCE node
I20260924 10:35:19.419744 9706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:19.420390 9706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:19.421563 9706 hybrid_clock.cc:648] HybridClock initialized: now 1790246119421553 us; error 44 us; skew 500 ppm
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:19.424607 9706 init.cc:377] Logged in from keytab as oryx/127.2.193.129@KRBTEST.COM (short username oryx)
I20260924 10:35:19.425540 9706 webserver.cc:460] Webserver started at http://127.2.193.129:46625/ using document root <none> and password file <none>
I20260924 10:35:19.426110 9706 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:19.426216 9706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:19.430151 9706 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:35:19.432282 9810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.433700 9706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:35:19.433823 9706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "ae8bb19345504f579f0b7e49c8e4f96e"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "1b27fe57822e95216adc1f0c51855222"
server_key_iv: "0addcb0a1dfb8529b16b42c9fdedcf65"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.434450 9706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:19.448372 9706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:19.451642 9706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:19.451938 9706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:19.452569 9706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:19.453481 9706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:19.453547 9706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.453614 9706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:19.453653 9706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.463511 9706 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38325
I20260924 10:35:19.463531 9923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38325 every 8 connection(s)
I20260924 10:35:19.464552 9706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:19.471549 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 9706
I20260924 10:35:19.471750 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 9156
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.189@KRBTEST.COM
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.188@KRBTEST.COM
I20260924 10:35:19.478713 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:36309
--local_ip_for_outbound_sockets=127.2.193.130
--tserver_master_addrs=127.2.193.190:40919
--webserver_port=40507
--webserver_interface=127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab
--principal=oryx/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:35:19.482396 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.466899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:42069 (local address 127.2.193.190:40919)
0924 10:35:19.467034 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.467038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.467667 (+ 629us) server_negotiation.cc:407] Connection header received
0924 10:35:19.468768 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.468772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.468824 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.468882 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.470283 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.471099 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.473930 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.474388 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.474518 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.477702 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.477706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.477709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.477743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.480377 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.480962 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.480967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.480970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.481065 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.481389 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.481392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.481394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.481597 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.481675 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.482102 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.482228 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:35:19.482402 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.466541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:43581 (local address 127.2.193.189:34413)
0924 10:35:19.467065 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.467073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.468057 (+ 984us) server_negotiation.cc:407] Connection header received
0924 10:35:19.468744 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.468747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.468800 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.468879 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.470245 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.471099 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.473477 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.473831 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.473921 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.477996 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.478003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.478009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.478044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.479846 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.480520 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.480524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.480526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.480611 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.481022 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.481028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.481032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.481310 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.481407 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.482102 (+ 695us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.482228 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260924 10:35:19.482519 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.466673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:39589 (local address 127.2.193.188:45491)
0924 10:35:19.467215 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.467225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.468097 (+ 872us) server_negotiation.cc:407] Connection header received
0924 10:35:19.468768 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.468772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.468824 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.468896 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.470277 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.470990 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.474727 (+ 3737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.475205 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.475336 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.478408 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.478412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.478415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.478447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.480269 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.480910 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.480915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.480919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.481011 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.481385 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.481389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.481391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.481632 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.481720 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.482094 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.482248 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":82,"threads_started":1}
I20260924 10:35:19.484186 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.466876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.2.193.129:39589)
0924 10:35:19.467956 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.467969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.468536 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.469060 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.469067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.469620 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.470114 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.470123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.471139 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.471141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.474610 (+ 3469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.474619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.475477 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.475484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.475618 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.476110 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.476130 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.478221 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.480429 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.480435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.480437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.480790 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.481134 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.481138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.481140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.481254 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.481780 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.481785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.481890 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.483023 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.483203 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":1811,"spinlock_wait_cycles":71680,"thread_start_us":167,"threads_started":1}
I20260924 10:35:19.484185 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.466885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.129:42069)
0924 10:35:19.467509 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.467564 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.468536 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.469060 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.469067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.469620 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.470116 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.470123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.471210 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.471213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.473794 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.473804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.474858 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.474866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.474967 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.475468 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.475482 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.477453 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.480515 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.480517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.480519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.480835 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.481200 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.481203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.481204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.481280 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.481725 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.481728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.481806 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.482623 (+ 817us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.483006 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":1686,"spinlock_wait_cycles":30848,"thread_start_us":118,"threads_started":1}
I20260924 10:35:19.484182 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.466876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.2.193.129:43581)
0924 10:35:19.467958 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.467970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.468545 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.469058 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.469066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.469563 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.470104 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.470118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.471537 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.471540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.473317 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.473328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.474108 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.474111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.474259 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.474924 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.474946 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.477835 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.479989 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.479996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.480016 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.480404 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.480725 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.480728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.480732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.480919 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.481472 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.481478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.481806 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.482623 (+ 817us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.483053 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":262,"spinlock_wait_cycles":5376,"thread_start_us":127,"threads_started":1}
I20260924 10:35:19.485934 9924 heartbeater.cc:347] Connected to a master server at 127.2.193.188:45491
I20260924 10:35:19.486132 9925 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40919
I20260924 10:35:19.486217 9924 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.486750 9924 heartbeater.cc:511] Master 127.2.193.188:45491 requested a full tablet report, sending...
I20260924 10:35:19.487231 9926 heartbeater.cc:347] Connected to a master server at 127.2.193.189:34413
I20260924 10:35:19.487301 9926 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.487437 9926 heartbeater.cc:511] Master 127.2.193.189:34413 requested a full tablet report, sending...
I20260924 10:35:19.486228 9925 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.487859 9925 heartbeater.cc:511] Master 127.2.193.190:40919 requested a full tablet report, sending...
I20260924 10:35:19.488386 9561 ts_manager.cc:194] Registered new tserver with Master: ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:19.488440 9745 ts_manager.cc:194] Registered new tserver with Master: ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:19.489629 8960 ts_manager.cc:194] Re-registered known tserver with Master: ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:19.490568 8960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:42069
I20260924 10:35:19.566293 9419 heartbeater.cc:503] Master 127.2.193.190:40919 was elected leader, sending a full tablet report...
W20260924 10:35:19.597205 9932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:19.597522 9932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:19.597570 9932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:19.601716 9932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:19.601783 9932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:19.601848 9932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:19.607358 9932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab
--principal=oryx/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:36309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=40507
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:19.608492 9932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:19.609608 9932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:19.619612 9940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.619622 9937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.619629 9938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:19.620028 9932 server_base.cc:1034] running on GCE node
I20260924 10:35:19.620440 9932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:19.621129 9932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:19.622324 9932 hybrid_clock.cc:648] HybridClock initialized: now 1790246119622306 us; error 33 us; skew 500 ppm
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:19.625046 9932 init.cc:377] Logged in from keytab as oryx/127.2.193.130@KRBTEST.COM (short username oryx)
I20260924 10:35:19.626004 9932 webserver.cc:460] Webserver started at http://127.2.193.130:40507/ using document root <none> and password file <none>
I20260924 10:35:19.626596 9932 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:19.626704 9932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:19.630673 9932 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 10:35:19.632938 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.634491 9932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:35:19.634614 9932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "ba188de6cd214f1cad09f370758fb39c"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "6812c79892f917727ff9a76bc7fcc074"
server_key_iv: "2fc139c19f5b4e8ed0a5d8c0a1f806f3"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.635249 9932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:19.654501 9932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:19.658206 9932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:19.658418 9932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:19.659036 9932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:19.659899 9932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:19.659945 9932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.660009 9932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:19.660039 9932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.668573 9932 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:36309
I20260924 10:35:19.668629 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:36309 every 8 connection(s)
I20260924 10:35:19.669641 9932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:19.676441 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 9932
I20260924 10:35:19.676620 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 9290
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.188@KRBTEST.COM
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.189@KRBTEST.COM
I20260924 10:35:19.685041 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:45253
--local_ip_for_outbound_sockets=127.2.193.131
--tserver_master_addrs=127.2.193.190:40919
--webserver_port=33629
--webserver_interface=127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab
--principal=oryx/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:35:19.686936 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.673664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:52175 (local address 127.2.193.190:40919)
0924 10:35:19.673794 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.673798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.673811 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:35:19.673851 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.673854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.673899 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.673977 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.676365 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.677133 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.678259 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.678490 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.678600 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.682068 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.682071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.682074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.682102 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.684361 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685022 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.685026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.685028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.685119 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685590 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.685593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.685595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.685832 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.685905 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.686625 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.686754 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:35:19.686969 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.671382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:48217 (local address 127.2.193.189:34413)
0924 10:35:19.671507 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.671515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.672673 (+ 1158us) server_negotiation.cc:407] Connection header received
0924 10:35:19.673826 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.673830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.673885 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.673941 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.675402 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.676226 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.679088 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.679529 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.679624 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.683207 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.683213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.683217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.683249 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.685238 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685720 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.685723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.685725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.685813 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.686115 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.686118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.686120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.686316 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.686396 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.686719 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.686843 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:35:19.687162 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.673081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:49565 (local address 127.2.193.188:45491)
0924 10:35:19.673317 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.673321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.673543 (+ 222us) server_negotiation.cc:407] Connection header received
0924 10:35:19.673794 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.673798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.673866 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.673921 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.675390 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.676228 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.678081 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.678521 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.678648 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.683219 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.683226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.683232 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.683267 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.685067 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685665 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.685672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.685677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.685765 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.686083 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.686089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.686094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.686330 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.686422 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.686778 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.686955 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260924 10:35:19.688735 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.672938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.2.193.130:49565)
0924 10:35:19.673460 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.673476 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.673609 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.674146 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.674154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.674616 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.675233 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.675246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.676387 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.676390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.677968 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.677978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.678743 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.678748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.678895 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.679721 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.679739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.683053 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.685225 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.685236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.685550 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.685886 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.685896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.685984 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.686487 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.686491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.686597 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.687351 (+ 754us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.687735 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":366,"spinlock_wait_cycles":12416,"thread_start_us":108,"threads_started":1}
I20260924 10:35:19.688735 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.673377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:52175)
0924 10:35:19.673639 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.673652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.673742 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.674147 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.674155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.674620 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.675235 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.675249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.677290 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.677294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.678151 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.678158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.678633 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.678637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.678861 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.679617 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.679643 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.681898 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.684504 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.684512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.684530 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.684891 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.685264 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685280 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.685284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.685472 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.685982 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.685988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.686337 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.687192 (+ 855us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.687574 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":232,"spinlock_wait_cycles":13824,"thread_start_us":94,"threads_started":1}
I20260924 10:35:19.689049 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.672095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.2.193.130:48217)
0924 10:35:19.672509 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.672551 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.673567 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.674147 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.674155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.674618 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.675235 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.675249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.676365 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.676368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.678946 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.678958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.679805 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.679811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.679939 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.680493 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.680514 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.683035 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.685342 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.685346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.685614 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.685923 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.685929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.685934 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.686017 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.686449 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.686451 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.686533 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.687106 (+ 573us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.687630 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":1115,"thread_start_us":115,"threads_started":1}
I20260924 10:35:19.690028 10062 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40919
I20260924 10:35:19.690061 10061 heartbeater.cc:347] Connected to a master server at 127.2.193.188:45491
I20260924 10:35:19.690287 10061 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.690311 10063 heartbeater.cc:347] Connected to a master server at 127.2.193.189:34413
I20260924 10:35:19.690378 10063 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.690287 10062 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.690929 10061 heartbeater.cc:511] Master 127.2.193.188:45491 requested a full tablet report, sending...
I20260924 10:35:19.690939 10062 heartbeater.cc:511] Master 127.2.193.190:40919 requested a full tablet report, sending...
I20260924 10:35:19.690948 10063 heartbeater.cc:511] Master 127.2.193.189:34413 requested a full tablet report, sending...
I20260924 10:35:19.692067 8960 ts_manager.cc:194] Re-registered known tserver with Master: ba188de6cd214f1cad09f370758fb39c (127.2.193.130:36309)
I20260924 10:35:19.692441 9745 ts_manager.cc:194] Registered new tserver with Master: ba188de6cd214f1cad09f370758fb39c (127.2.193.130:36309)
I20260924 10:35:19.692843 8960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:52175
I20260924 10:35:19.693373 9561 ts_manager.cc:194] Registered new tserver with Master: ba188de6cd214f1cad09f370758fb39c (127.2.193.130:36309)
I20260924 10:35:19.713379 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.709747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.0.0.1:36340)
0924 10:35:19.709887 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.709905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.709995 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.710313 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.710315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.710330 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:19.710412 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.710417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.711401 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.711404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.712132 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.712141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.712996 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.712998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.713061 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.713081 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.713128 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.713240 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:35:19.713390 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.709814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.2.193.188:45491)
0924 10:35:19.709951 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.709956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.709974 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:35:19.710098 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.710101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.710162 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.710268 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:19.710514 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.711281 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.712244 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.712879 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.712989 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.713139 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.713201 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.713267 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:35:19.714190 9755 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 10:35:19.714658 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f037e6aa20415a9226a44178f67e35. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.714820 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: This master's current role is: LEARNER
W20260924 10:35:19.803974 10068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:19.804227 10068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:19.804270 10068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:19.808403 10068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:19.808466 10068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:19.808525 10068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:19.815855 10068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab
--principal=oryx/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:45253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=33629
--tserver_master_addrs=127.2.193.190:40919,127.2.193.189:34413,127.2.193.188:45491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:19.817109 10068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:19.818372 10068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:19.828685 10073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:19.828764 10068 server_base.cc:1034] running on GCE node
W20260924 10:35:19.828661 10076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:19.828702 10074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:19.829423 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:19.830085 10068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:19.831264 10068 hybrid_clock.cc:648] HybridClock initialized: now 1790246119831232 us; error 42 us; skew 500 ppm
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:19.834033 10068 init.cc:377] Logged in from keytab as oryx/127.2.193.131@KRBTEST.COM (short username oryx)
I20260924 10:35:19.834949 10068 webserver.cc:460] Webserver started at http://127.2.193.131:33629/ using document root <none> and password file <none>
I20260924 10:35:19.835515 10068 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:19.835605 10068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:19.839821 10068 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:35:19.841924 10083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.843184 10068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:19.843312 10068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "7b504c9120404b93a9f5752c140b8013"
format_stamp: "Formatted at 2026-09-24 10:35:18 on dist-test-slave-8wc7"
server_key: "43b82ea2943d2dacfb92e4f2ec3b882b"
server_key_iv: "7fb5b47d15ebfc8b6c65078bf7a8be7c"
server_key_version: "encryptionkey@0"
I20260924 10:35:19.843844 10068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:19.861060 10068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:19.863941 10068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:19.864158 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:19.864765 10068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:19.865698 10068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:19.865767 10068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.865836 10068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:19.865875 10068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:19.875083 10068 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:45253
I20260924 10:35:19.875131 10196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:45253 every 8 connection(s)
I20260924 10:35:19.876137 10068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:19.884714 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 10068
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.189@KRBTEST.COM
Sep 24 10:35:19 dist-test-slave-8wc7 krb5kdc[8915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246119, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.188@KRBTEST.COM
I20260924 10:35:19.892396 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.878532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:60173 (local address 127.2.193.190:40919)
0924 10:35:19.878658 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.878662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.879307 (+ 645us) server_negotiation.cc:407] Connection header received
0924 10:35:19.880325 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.880329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.880384 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.880436 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.881826 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.882625 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.884122 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.884406 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.884497 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.888040 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.888045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.888048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.888081 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.890004 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.890643 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.890646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.890648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.890732 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.891160 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.891163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.891165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.891356 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.891440 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.892113 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.892230 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:35:19.892437 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.878147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:45225 (local address 127.2.193.189:34413)
0924 10:35:19.878251 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.878255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.880000 (+ 1745us) server_negotiation.cc:407] Connection header received
0924 10:35:19.880325 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.880329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.880384 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.880436 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.881826 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.882625 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.884102 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.884399 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.884517 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.888088 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.888091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.888094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.888119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.890437 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.891046 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.891050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.891051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.891138 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.891447 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.891450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.891453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.891671 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.891742 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.892141 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.892262 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 10:35:19.893007 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.877981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:33351 (local address 127.2.193.188:45491)
0924 10:35:19.878113 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.878118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.879762 (+ 1644us) server_negotiation.cc:407] Connection header received
0924 10:35:19.880287 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.880290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.880361 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.880458 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.881801 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.882321 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.883467 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.883679 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.883782 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.889100 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.889103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.889105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.889129 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.891247 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.891726 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.891730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.891731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.891814 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.892117 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.892120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.892121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.892319 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.892399 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.892748 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.892859 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:35:19.894173 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.878586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.131:60173)
0924 10:35:19.879147 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.879200 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.880095 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.880630 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.880637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.881118 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.881674 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.881682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.882768 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.882771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.883989 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.883998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.884670 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.884673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.884737 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.885286 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.885308 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.887833 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.890137 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.890146 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.890163 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.890538 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.890849 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.890855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.890860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.891052 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.891492 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.891500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.891819 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.892933 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.893312 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":245,"spinlock_wait_cycles":18816,"thread_start_us":177,"threads_started":1}
I20260924 10:35:19.894174 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.878586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.2.193.131:45225)
0924 10:35:19.879917 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.879927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.880095 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.880630 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.880637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.881141 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.881676 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.881682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.882768 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.882771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.883989 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.883998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.884645 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.884649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.884778 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.885391 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.885405 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.887932 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.890684 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.890687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.890689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.890922 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.891251 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.891254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.891255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.891341 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.891810 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.891813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.891933 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.892651 (+ 718us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.893003 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":375,"spinlock_wait_cycles":46976,"thread_start_us":143,"threads_started":1}
I20260924 10:35:19.894423 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.878586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.2.193.131:33351)
0924 10:35:19.879666 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.879677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.880096 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.880628 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.880635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.881072 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:19.881665 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.881678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.882463 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.882466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.883339 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.883348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.883820 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.883824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.883999 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.887496 (+ 3497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:19.887510 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:19.888960 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:19.891366 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.891372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.891377 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.891613 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.891919 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:19.891922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:19.891925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:19.892020 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:19.892476 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:19.892479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:19.892569 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.893197 (+ 628us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.893381 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":142,"spinlock_wait_cycles":77952,"thread_start_us":181,"threads_started":1}
I20260924 10:35:19.895824 10197 heartbeater.cc:347] Connected to a master server at 127.2.193.188:45491
I20260924 10:35:19.895805 10199 heartbeater.cc:347] Connected to a master server at 127.2.193.189:34413
I20260924 10:35:19.895803 10198 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40919
I20260924 10:35:19.896081 10198 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.896083 10199 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.896081 10197 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:19.896683 10197 heartbeater.cc:511] Master 127.2.193.188:45491 requested a full tablet report, sending...
I20260924 10:35:19.896683 10199 heartbeater.cc:511] Master 127.2.193.189:34413 requested a full tablet report, sending...
I20260924 10:35:19.897251 10198 heartbeater.cc:511] Master 127.2.193.190:40919 requested a full tablet report, sending...
I20260924 10:35:19.897914 9745 ts_manager.cc:194] Registered new tserver with Master: 7b504c9120404b93a9f5752c140b8013 (127.2.193.131:45253)
I20260924 10:35:19.897907 9561 ts_manager.cc:194] Registered new tserver with Master: 7b504c9120404b93a9f5752c140b8013 (127.2.193.131:45253)
I20260924 10:35:19.897907 8960 ts_manager.cc:194] Re-registered known tserver with Master: 7b504c9120404b93a9f5752c140b8013 (127.2.193.131:45253)
I20260924 10:35:19.898643 8960 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:60173
I20260924 10:35:19.903136 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.886976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.2.193.190:40919)
0924 10:35:19.888390 (+ 1414us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.888395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.888817 (+ 422us) server_negotiation.cc:407] Connection header received
0924 10:35:19.888864 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.888869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.888924 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.889004 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.890918 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.891689 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.894725 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.895394 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.895493 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.899767 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.899771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.899774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.899799 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.901460 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.901867 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.901871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.901873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.901960 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.902235 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.902242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.902246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.902471 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.902601 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.902824 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.902957 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":63,"threads_started":1}
I20260924 10:35:19.903789 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.887252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.2.193.189:34413)
0924 10:35:19.887498 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.887502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.887995 (+ 493us) server_negotiation.cc:407] Connection header received
0924 10:35:19.888039 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.888044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.888095 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.888173 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.892857 (+ 4684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.893634 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.895618 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.895966 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.896038 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.900033 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.900036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.900039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.900065 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.901874 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.902295 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.902298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.902300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.902387 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.902669 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.902674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.902675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.902885 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.903007 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.903580 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.903674 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260924 10:35:19.909173 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.886984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.2.193.188:45491)
0924 10:35:19.887651 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.887656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.900244 (+ 12588us) server_negotiation.cc:407] Connection header received
0924 10:35:19.900434 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.900439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.900514 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.900578 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:19.901499 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.902279 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.903314 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.903638 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.903696 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.905921 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:19.905925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:19.905927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:19.905952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:19.907620 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.908036 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.908040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.908041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.908124 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:19.908409 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:19.908412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:19.908413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:19.908580 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:19.908679 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.908904 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.908998 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":88,"threads_started":1}
I20260924 10:35:19.906608 8960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:19.910800 8960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:19.913259 9521 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: NON_VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: true } } }
I20260924 10:35:19.913398 9521 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:19.916462 9521 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: attempting to promote NON_VOTER 41eb1f74cba6410d98bd5768c7626cfb to VOTER
I20260924 10:35:19.916807 9521 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } }
I20260924 10:35:19.918417 9754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 LEARNER]: Refusing update from remote peer e3f037e6aa20415a9226a44178f67e35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 10:35:19.918813 9571 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Refusing update from remote peer e3f037e6aa20415a9226a44178f67e35: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 10:35:19.918817 10215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 10:35:19.919322 10215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 10:35:19.920342 10216 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.921453 9571 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.921940 10216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.922031 10216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:19.921912 10215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.922127 10216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.922189 10215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:19.922210 10216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35 [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:19.922696 9754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.923666 10225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.923800 10225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:35:19.924829 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3f037e6aa20415a9226a44178f67e35" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:19.924947 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:35:19.932808 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.926306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36309 (local address 127.0.0.1:55846)
0924 10:35:19.926624 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.926639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.926720 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.928014 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.928017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.928034 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:19.928136 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.928142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.929102 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.929106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.930358 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.930371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.932434 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.932437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.932518 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.932539 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.932600 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.932676 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2}
I20260924 10:35:19.932876 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.926602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.2.193.130:36309)
0924 10:35:19.927500 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.927515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.927541 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:35:19.927625 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.927631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.927884 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.928050 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:19.928233 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.928988 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.931634 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.932311 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.932435 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.932592 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.932645 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.932732 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":113,"threads_started":1}
I20260924 10:35:19.933334 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.926603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38325 (local address 127.0.0.1:53996)
0924 10:35:19.926908 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.926920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.926998 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.929035 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.929039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.929064 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:19.929268 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.929277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.930237 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.930244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.931491 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.931501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.932573 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.932577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.932663 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.932684 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.932734 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.932801 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":81,"thread_start_us":122,"threads_started":1}
I20260924 10:35:19.933401 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.927005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.2.193.129:38325)
0924 10:35:19.928684 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.928692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.928709 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:35:19.928766 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.928769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.928905 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.929013 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:19.929415 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.930122 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.931611 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.932432 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.932936 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.933084 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.933140 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.933264 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1587,"thread_start_us":95,"threads_started":1}
I20260924 10:35:19.933450 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.926674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:45253 (local address 127.0.0.1:46732)
0924 10:35:19.927162 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:19.927173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:19.927245 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:19.928366 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:19.928370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:19.928385 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:19.928468 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.928474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.930460 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.930464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.931585 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:19.931593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.932994 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.932999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:19.933092 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.933117 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:35:19.933214 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:35:19.933285 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":99,"threads_started":1}
I20260924 10:35:19.934314 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:19.927159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.2.193.131:45253)
0924 10:35:19.927749 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:19.927762 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:19.927784 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:35:19.928017 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:19.928023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:19.928225 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:19.928393 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:19.929175 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.930338 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.931690 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:19.932431 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:19.932552 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:19.932687 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:19.934068 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0924 10:35:19.934173 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":93,"threads_started":1}
I20260924 10:35:19.936072 9995 tablet_service.cc:1467] Processing CreateTablet for tablet d705f1b1a3404c56b546a0bce53c2c7b (DEFAULT_TABLE table=test-table [id=b1e9ce5efef14a14a37e856b88be3070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:19.936419 9858 tablet_service.cc:1467] Processing CreateTablet for tablet d705f1b1a3404c56b546a0bce53c2c7b (DEFAULT_TABLE table=test-table [id=b1e9ce5efef14a14a37e856b88be3070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:19.936836 10131 tablet_service.cc:1467] Processing CreateTablet for tablet d705f1b1a3404c56b546a0bce53c2c7b (DEFAULT_TABLE table=test-table [id=b1e9ce5efef14a14a37e856b88be3070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:19.937263 9995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d705f1b1a3404c56b546a0bce53c2c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:19.937263 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d705f1b1a3404c56b546a0bce53c2c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:19.937762 10131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d705f1b1a3404c56b546a0bce53c2c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:19.943986 10231 tablet_bootstrap.cc:492] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Bootstrap starting.
I20260924 10:35:19.945240 10232 tablet_bootstrap.cc:492] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Bootstrap starting.
I20260924 10:35:19.947775 10232 tablet_bootstrap.cc:654] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:19.947775 10231 tablet_bootstrap.cc:654] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:19.948684 10232 log.cc:824] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:19.948684 10231 log.cc:824] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:19.950850 10231 tablet_bootstrap.cc:492] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: No bootstrap required, opened a new log
I20260924 10:35:19.950850 10232 tablet_bootstrap.cc:492] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: No bootstrap required, opened a new log
I20260924 10:35:19.951046 10231 ts_tablet_manager.cc:1397] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:35:19.951046 10232 ts_tablet_manager.cc:1397] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:35:19.949455 10233 tablet_bootstrap.cc:492] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Bootstrap starting.
I20260924 10:35:19.955204 10233 tablet_bootstrap.cc:654] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:19.955761 10232 raft_consensus.cc:348] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:19.955761 10231 raft_consensus.cc:348] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:19.956048 10232 raft_consensus.cc:374] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:19.956048 10231 raft_consensus.cc:374] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:19.956100 10231 raft_consensus.cc:729] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8bb19345504f579f0b7e49c8e4f96e, State: Initialized, Role: FOLLOWER
I20260924 10:35:19.956100 10232 raft_consensus.cc:729] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba188de6cd214f1cad09f370758fb39c, State: Initialized, Role: FOLLOWER
I20260924 10:35:19.956080 10233 log.cc:824] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:19.956606 10232 consensus_queue.cc:260] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:19.956606 10231 consensus_queue.cc:260] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:19.957347 9925 heartbeater.cc:503] Master 127.2.193.190:40919 was elected leader, sending a full tablet report...
I20260924 10:35:19.957659 10231 ts_tablet_manager.cc:1428] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 10:35:19.957659 10232 ts_tablet_manager.cc:1428] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 10:35:19.959100 10062 heartbeater.cc:503] Master 127.2.193.190:40919 was elected leader, sending a full tablet report...
I20260924 10:35:19.962335 10233 tablet_bootstrap.cc:492] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: No bootstrap required, opened a new log
I20260924 10:35:19.962564 10233 ts_tablet_manager.cc:1397] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
W20260924 10:35:19.965984 9927 tablet.cc:2369] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:35:19.966676 10233 raft_consensus.cc:348] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:19.966892 10233 raft_consensus.cc:374] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:19.966965 10233 raft_consensus.cc:729] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b504c9120404b93a9f5752c140b8013, State: Initialized, Role: FOLLOWER
I20260924 10:35:19.967383 10233 consensus_queue.cc:260] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:19.968052 10198 heartbeater.cc:503] Master 127.2.193.190:40919 was elected leader, sending a full tablet report...
I20260924 10:35:19.968366 10233 ts_tablet_manager.cc:1428] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:35:20.017773 10239 raft_consensus.cc:482] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:20.018019 10239 raft_consensus.cc:504] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:20.019217 10239 leader_election.cc:288] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba188de6cd214f1cad09f370758fb39c (127.2.193.130:36309), ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:20.022805 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.019543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:35189 (local address 127.2.193.129:38325)
0924 10:35:20.019661 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:20.019668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:20.019743 (+ 75us) server_negotiation.cc:407] Connection header received
0924 10:35:20.019890 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:20.019892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:20.019932 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:20.020012 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:20.020267 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.020752 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.021853 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.022246 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.022369 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.022423 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:20.022586 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 10:35:20.022659 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:35:20.022826 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.019623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:35867 (local address 127.2.193.130:36309)
0924 10:35:20.019754 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:20.019758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:20.019769 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:35:20.019812 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:20.019814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:20.019852 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:20.019939 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:20.020195 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.020697 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.021876 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.022277 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.022413 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.022465 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:20.022623 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:35:20.022699 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:35:20.022861 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.019423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36309 (local address 127.2.193.131:35867)
0924 10:35:20.019528 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:20.019545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:20.019638 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:20.019952 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:20.019955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:20.019970 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:20.020088 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:20.020093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.020814 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.020818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:20.021748 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:20.021758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.022400 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.022404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:20.022503 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.022520 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:35:20.022598 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:35:20.022697 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":68}
I20260924 10:35:20.022867 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.019483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38325 (local address 127.2.193.131:35189)
0924 10:35:20.019659 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:20.019673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:20.019782 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:20.020031 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:20.020034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:20.020051 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:20.020165 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:20.020170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.020853 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.020859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:20.021735 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:20.021744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.022359 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.022362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:20.022456 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.022491 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:35:20.022573 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:35:20.022659 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":41}
I20260924 10:35:20.023530 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d705f1b1a3404c56b546a0bce53c2c7b" candidate_uuid: "7b504c9120404b93a9f5752c140b8013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba188de6cd214f1cad09f370758fb39c" is_pre_election: true
I20260924 10:35:20.023633 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d705f1b1a3404c56b546a0bce53c2c7b" candidate_uuid: "7b504c9120404b93a9f5752c140b8013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" is_pre_election: true
I20260924 10:35:20.023862 10015 raft_consensus.cc:2454] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b504c9120404b93a9f5752c140b8013 in term 0.
I20260924 10:35:20.023918 9878 raft_consensus.cc:2454] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b504c9120404b93a9f5752c140b8013 in term 0.
I20260924 10:35:20.024372 10087 leader_election.cc:302] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7b504c9120404b93a9f5752c140b8013, ae8bb19345504f579f0b7e49c8e4f96e, ba188de6cd214f1cad09f370758fb39c; no voters:
I20260924 10:35:20.024665 10239 raft_consensus.cc:2781] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:20.024763 10239 raft_consensus.cc:482] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:20.024813 10239 raft_consensus.cc:3037] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:20.025934 10239 raft_consensus.cc:504] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:20.026294 10239 leader_election.cc:288] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [CANDIDATE]: Term 1 election: Requested vote from peers ba188de6cd214f1cad09f370758fb39c (127.2.193.130:36309), ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:20.026629 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d705f1b1a3404c56b546a0bce53c2c7b" candidate_uuid: "7b504c9120404b93a9f5752c140b8013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bb19345504f579f0b7e49c8e4f96e"
I20260924 10:35:20.026628 10015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d705f1b1a3404c56b546a0bce53c2c7b" candidate_uuid: "7b504c9120404b93a9f5752c140b8013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba188de6cd214f1cad09f370758fb39c"
I20260924 10:35:20.026758 10015 raft_consensus.cc:3037] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:20.026758 9878 raft_consensus.cc:3037] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:20.028005 9878 raft_consensus.cc:2454] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b504c9120404b93a9f5752c140b8013 in term 1.
I20260924 10:35:20.028005 10015 raft_consensus.cc:2454] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b504c9120404b93a9f5752c140b8013 in term 1.
I20260924 10:35:20.028281 10085 leader_election.cc:302] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b504c9120404b93a9f5752c140b8013, ba188de6cd214f1cad09f370758fb39c; no voters:
I20260924 10:35:20.028486 10239 raft_consensus.cc:2781] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:20.028705 10239 raft_consensus.cc:686] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 1 LEADER]: Becoming Leader. State: Replica: 7b504c9120404b93a9f5752c140b8013, State: Running, Role: LEADER
I20260924 10:35:20.029079 10239 consensus_queue.cc:237] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } }
I20260924 10:35:20.032603 8959 catalog_manager.cc:5495] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b504c9120404b93a9f5752c140b8013 (127.2.193.131). New cstate: current_term: 1 leader_uuid: "7b504c9120404b93a9f5752c140b8013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } health_report { overall_health: UNKNOWN } } }
I20260924 10:35:20.062108 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.058768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.2.193.131:45253)
0924 10:35:20.058894 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:20.058897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:20.058912 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:35:20.059006 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:20.059008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:20.059063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:20.059151 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:20.059365 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.059889 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.060699 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.060898 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.060972 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.061252 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:20.061684 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:20.061774 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:20.061875 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 10:35:20.061925 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:35:20.067930 10015 raft_consensus.cc:1261] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 1 FOLLOWER]: Refusing update from remote peer 7b504c9120404b93a9f5752c140b8013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:20.068607 9878 raft_consensus.cc:1261] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Refusing update from remote peer 7b504c9120404b93a9f5752c140b8013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:20.068753 10239 consensus_queue.cc:1035] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:20.069274 10239 consensus_queue.cc:1035] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:20.080371 10244 mvcc.cc:205] Tried to move back new op lower bound from 7332848107791024128 to 7332848107642294272. Current Snapshot: MvccSnapshot[applied={T|T < 7332848107791024128}]
I20260924 10:35:20.082149 10245 mvcc.cc:205] Tried to move back new op lower bound from 7332848107791024128 to 7332848107642294272. Current Snapshot: MvccSnapshot[applied={T|T < 7332848107791024128 or (T in {7332848107791024128})}]
I20260924 10:35:20.089987 8959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:20.090180 8959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:20.094713 8959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f037e6aa20415a9226a44178f67e35: Sending DeleteTablet for 3 replicas of tablet d705f1b1a3404c56b546a0bce53c2c7b
I20260924 10:35:20.095534 9858 tablet_service.cc:1514] Processing DeleteTablet for tablet d705f1b1a3404c56b546a0bce53c2c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:20 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:53996
I20260924 10:35:20.095589 10131 tablet_service.cc:1514] Processing DeleteTablet for tablet d705f1b1a3404c56b546a0bce53c2c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:20 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:46732
I20260924 10:35:20.095968 9995 tablet_service.cc:1514] Processing DeleteTablet for tablet d705f1b1a3404c56b546a0bce53c2c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:20 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:55846
I20260924 10:35:20.096073 10253 tablet_replica.cc:330] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: stopping tablet replica
I20260924 10:35:20.096403 10253 raft_consensus.cc:2229] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:20.096680 10255 tablet_replica.cc:330] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: stopping tablet replica
I20260924 10:35:20.096764 10253 raft_consensus.cc:2258] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:20.097076 10255 raft_consensus.cc:2229] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:20.097483 10254 tablet_replica.cc:330] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: stopping tablet replica
I20260924 10:35:20.097479 10255 raft_consensus.cc:2258] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:20.097754 10254 raft_consensus.cc:2229] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:20.098160 10254 raft_consensus.cc:2258] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:20.099246 10255 ts_tablet_manager.cc:1905] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:20.099560 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 8928
I20260924 10:35:20.100904 10253 ts_tablet_manager.cc:1905] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:20.102941 10255 ts_tablet_manager.cc:1918] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:20.103120 10255 log.cc:1196] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/d705f1b1a3404c56b546a0bce53c2c7b
I20260924 10:35:20.103503 10253 ts_tablet_manager.cc:1918] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:20.103602 10253 log.cc:1196] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/d705f1b1a3404c56b546a0bce53c2c7b
I20260924 10:35:20.103534 10255 ts_tablet_manager.cc:1939] T d705f1b1a3404c56b546a0bce53c2c7b P ba188de6cd214f1cad09f370758fb39c: Deleting consensus metadata
I20260924 10:35:20.103907 10253 ts_tablet_manager.cc:1939] T d705f1b1a3404c56b546a0bce53c2c7b P ae8bb19345504f579f0b7e49c8e4f96e: Deleting consensus metadata
I20260924 10:35:20.106155 10254 ts_tablet_manager.cc:1905] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:35:20.107944 9811 connection.cc:535] server connection from 127.0.0.1:53996 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 10:35:20.108340 9949 connection.cc:535] server connection from 127.0.0.1:55846 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:35:20.109712 10254 ts_tablet_manager.cc:1918] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:20.109891 10254 log.cc:1196] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/d705f1b1a3404c56b546a0bce53c2c7b
I20260924 10:35:20.110291 10254 ts_tablet_manager.cc:1939] T d705f1b1a3404c56b546a0bce53c2c7b P 7b504c9120404b93a9f5752c140b8013: Deleting consensus metadata
W20260924 10:35:20.111546 10087 connection.cc:441] server connection from 127.0.0.1:46732 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46732 (request call id 1) could send its response
I20260924 10:35:20.115898 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.098778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.2.193.189:34413)
0924 10:35:20.098935 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:20.098939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:20.105286 (+ 6347us) server_negotiation.cc:407] Connection header received
0924 10:35:20.105498 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:20.105501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:20.105559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:20.105612 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:20.106606 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.107468 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.108784 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.109141 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.109253 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.111681 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:20.111685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:20.111688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:20.111716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:20.113615 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.114161 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.114165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.114167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.114259 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.114558 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.114561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.114563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.114757 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:20.114874 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:20.115622 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 10:35:20.115741 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:35:20.122648 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.113454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.2.193.189:34413)
0924 10:35:20.113573 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:20.113577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:20.113927 (+ 350us) server_negotiation.cc:407] Connection header received
0924 10:35:20.113962 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:20.113965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:20.114005 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:20.114067 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:20.114821 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.115568 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.116826 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.117156 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.117278 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.118958 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:20.118962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:20.118965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:20.118989 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:20.120672 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.121125 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.121128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.121130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.121245 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.121563 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.121566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.121568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.121756 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:20.121871 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:20.122368 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 10:35:20.122486 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:35:20.124157 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.112504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.2.193.188:45491)
0924 10:35:20.112647 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:20.112653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:20.112959 (+ 306us) server_negotiation.cc:407] Connection header received
0924 10:35:20.113135 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:20.113138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:20.113219 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:20.113305 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:20.114218 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.115010 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.118182 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.118519 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.118612 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.120259 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:20.120263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:20.120266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:20.120290 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:20.122074 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.122566 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.122571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.122573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.122656 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.122995 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.122998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.123000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.123223 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:20.123353 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:20.123927 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 10:35:20.124039 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:35:20.125531 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:20.098901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.2.193.188:45491)
0924 10:35:20.099042 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:20.099047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:20.116085 (+ 17038us) server_negotiation.cc:407] Connection header received
0924 10:35:20.116133 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:20.116136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:20.116181 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:20.116245 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:20.117311 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.118046 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.119594 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:20.119997 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:20.120099 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:20.121941 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:20.121945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:20.121951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:20.121981 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:20.123903 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.124328 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.124332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.124334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.124414 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:20.124702 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:20.124705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:20.124707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:20.124891 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:20.125027 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:20.125260 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 10:35:20.125371 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:35:21.080912 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.080340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.131:33397)
0924 10:35:21.080687 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.080793 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260924 10:35:21.080902 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.080336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.129:41443)
0924 10:35:21.080655 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.080761 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
W20260924 10:35:21.081241 10198 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:40919 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
W20260924 10:35:21.081305 9925 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:40919 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
I20260924 10:35:21.083037 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.082426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:34355)
0924 10:35:21.082771 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.082887 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":92,"threads_started":1}
W20260924 10:35:21.083318 10062 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:40919 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
I20260924 10:35:21.594733 10266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3f037e6aa20415a9226a44178f67e35)
I20260924 10:35:21.594882 10266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } }
I20260924 10:35:21.595322 10266 leader_election.cc:288] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919), f7028b32e8a9475ba90d969a51eb71ed (127.2.193.189:34413)
I20260924 10:35:21.595849 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3f037e6aa20415a9226a44178f67e35)
I20260924 10:35:21.596007 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.595542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42290)
0924 10:35:21.595797 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.595897 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
W20260924 10:35:21.596221 9732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
I20260924 10:35:21.596063 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } }
I20260924 10:35:21.597057 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.596819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42292)
0924 10:35:21.596929 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.596991 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:35:21.597164 9732 leader_election.cc:334] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919): Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
I20260924 10:35:21.597746 10267 leader_election.cc:288] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919), 41eb1f74cba6410d98bd5768c7626cfb (127.2.193.188:45491)
I20260924 10:35:21.598268 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.597812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42304)
0924 10:35:21.598091 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.598173 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":59,"threads_started":1}
W20260924 10:35:21.598429 9548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
I20260924 10:35:21.599265 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.599038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42320)
0924 10:35:21.599137 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.599181 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:35:21.599360 9548 leader_election.cc:334] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919): Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
I20260924 10:35:21.600261 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.595733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.2.193.189:34413)
0924 10:35:21.596144 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.596149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.596173 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:35:21.596219 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.596221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.596260 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.596377 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:21.596655 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.597462 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.599234 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.599792 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.599877 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.599924 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.600133 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.600185 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":62,"threads_started":1}
I20260924 10:35:21.600315 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.595542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:34413 (local address 127.0.0.1:53152)
0924 10:35:21.595859 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.595877 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.596012 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.596400 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:21.596403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:21.596423 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:21.596533 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.596539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.597576 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.597581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.599076 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.599085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.599913 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.599916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.600008 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.600034 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:35:21.600114 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:35:21.600184 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":27,"thread_start_us":63,"threads_started":1}
I20260924 10:35:21.600780 9571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41eb1f74cba6410d98bd5768c7626cfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7028b32e8a9475ba90d969a51eb71ed" is_pre_election: true
I20260924 10:35:21.600962 9571 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41eb1f74cba6410d98bd5768c7626cfb in term 1.
I20260924 10:35:21.601281 9729 leader_election.cc:302] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41eb1f74cba6410d98bd5768c7626cfb, f7028b32e8a9475ba90d969a51eb71ed; no voters: e3f037e6aa20415a9226a44178f67e35
I20260924 10:35:21.601451 10266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 10:35:21.601528 10266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3f037e6aa20415a9226a44178f67e35)
I20260924 10:35:21.601572 10266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:35:21.601979 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.597956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:45491 (local address 127.0.0.1:41572)
0924 10:35:21.598204 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.598217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.598297 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.598533 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:21.598536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:21.598557 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:21.598640 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.598645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.599587 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.599593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.600753 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.600762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.601678 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.601681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.601757 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.601775 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:35:21.601843 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:35:21.601893 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":64,"threads_started":1}
I20260924 10:35:21.602058 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.598072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.2.193.188:45491)
0924 10:35:21.598294 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.598303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.598320 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:35:21.598366 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.598369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.598409 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.598513 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:21.598761 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.599485 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.600863 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.601564 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.601672 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.601736 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.601874 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.601945 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260924 10:35:21.602753 10266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } }
I20260924 10:35:21.603102 10266 leader_election.cc:288] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 2 election: Requested vote from peers e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919), f7028b32e8a9475ba90d969a51eb71ed (127.2.193.189:34413)
I20260924 10:35:21.603498 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.603177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42328)
0924 10:35:21.603375 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.603432 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 10:35:21.603477 9571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41eb1f74cba6410d98bd5768c7626cfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7028b32e8a9475ba90d969a51eb71ed"
I20260924 10:35:21.603479 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7028b32e8a9475ba90d969a51eb71ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41eb1f74cba6410d98bd5768c7626cfb" is_pre_election: true
I20260924 10:35:21.603607 9571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:35:21.603729 9754 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7028b32e8a9475ba90d969a51eb71ed in current term 2: Already voted for candidate 41eb1f74cba6410d98bd5768c7626cfb in this term.
I20260924 10:35:21.604209 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.603998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42334)
0924 10:35:21.604103 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.604153 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:35:21.604300 9732 leader_election.cc:334] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919): Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
I20260924 10:35:21.604305 9547 leader_election.cc:302] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7028b32e8a9475ba90d969a51eb71ed; no voters: 41eb1f74cba6410d98bd5768c7626cfb, e3f037e6aa20415a9226a44178f67e35
I20260924 10:35:21.604820 9571 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41eb1f74cba6410d98bd5768c7626cfb in term 2.
I20260924 10:35:21.604929 10267 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:35:21.605118 9729 leader_election.cc:302] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41eb1f74cba6410d98bd5768c7626cfb, f7028b32e8a9475ba90d969a51eb71ed; no voters: e3f037e6aa20415a9226a44178f67e35
I20260924 10:35:21.605343 10266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:35:21.605562 10266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [term 2 LEADER]: Becoming Leader. State: Replica: 41eb1f74cba6410d98bd5768c7626cfb, State: Running, Role: LEADER
I20260924 10:35:21.605813 10266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } }
I20260924 10:35:21.607137 10276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41eb1f74cba6410d98bd5768c7626cfb. Latest consensus state: current_term: 2 leader_uuid: "41eb1f74cba6410d98bd5768c7626cfb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:21.607302 10276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:21.607573 10278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:21.609822 10278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:21.610271 10278 catalog_manager.cc:1259] Loaded cluster ID: 1a1fb670992b47c594a3927d6bcc75f5
I20260924 10:35:21.610332 10278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:21.610946 10278 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:21.611415 10278 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Loaded TSK: 0
I20260924 10:35:21.612076 10278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:35:21.650648 9745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:21.653921 9745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:21.657312 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.657026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42350)
0924 10:35:21.657129 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.657222 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 10:35:21.657451 9571 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [term 2 FOLLOWER]: Refusing update from remote peer 41eb1f74cba6410d98bd5768c7626cfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 10:35:21.658030 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.657771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.0.0.1:42358)
0924 10:35:21.657911 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.657968 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 10:35:21.657951 10266 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 10:35:21.658243 9732 consensus_peers.cc:487] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb -> Peer e3f037e6aa20415a9226a44178f67e35 (127.2.193.190:40919): Couldn't send request to peer e3f037e6aa20415a9226a44178f67e35. Status: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:35:21.659404 10267 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41eb1f74cba6410d98bd5768c7626cfb. Latest consensus state: current_term: 2 leader_uuid: "41eb1f74cba6410d98bd5768c7626cfb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:21.659533 10267 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:35:21.660521 10266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "41eb1f74cba6410d98bd5768c7626cfb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:21.660651 10266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:21.661455 10267 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "41eb1f74cba6410d98bd5768c7626cfb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3f037e6aa20415a9226a44178f67e35" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40919 } } peers { permanent_uuid: "f7028b32e8a9475ba90d969a51eb71ed" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 34413 } attrs { promote: false } } peers { permanent_uuid: "41eb1f74cba6410d98bd5768c7626cfb" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 45491 } attrs { promote: false } } }
I20260924 10:35:21.661615 10267 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7028b32e8a9475ba90d969a51eb71ed [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:35:21.670258 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.666252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:45253 (local address 127.0.0.1:38794)
0924 10:35:21.666365 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.666380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.666456 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.666847 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:21.666851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:21.666867 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:21.666960 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.666974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.667842 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.667848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.668996 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.669004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.669845 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.669849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.669937 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.669962 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:35:21.670035 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:35:21.670106 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 10:35:21.670734 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.666397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.2.193.131:45253)
0924 10:35:21.666639 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.666642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.666655 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:35:21.666697 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.666699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.666740 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.666838 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:21.667080 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.667696 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.669093 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.669719 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.669830 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.669898 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.670477 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.670571 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260924 10:35:21.671337 10131 tablet_service.cc:1467] Processing CreateTablet for tablet c536933936f84b9bbde360e081cdbfd7 (DEFAULT_TABLE table=test-table [id=c1b5ff10ba8341a2a01e15dfe493b301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:21.671602 10131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c536933936f84b9bbde360e081cdbfd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:21.672166 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.666832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36309 (local address 127.0.0.1:55220)
0924 10:35:21.667199 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.667215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.667319 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.667558 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:21.667561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:21.667580 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:21.667654 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.667660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.668635 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.668638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.669548 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.669556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.670707 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.670711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.670814 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.670839 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:35:21.670905 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:35:21.672077 (+ 1172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":1100}
I20260924 10:35:21.673292 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.667791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38325 (local address 127.0.0.1:57844)
0924 10:35:21.668039 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.668053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.668129 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.668634 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:21.668639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:21.668654 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:21.668728 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.668733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.670814 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.670817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.671953 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.671963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.672858 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.672861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.672949 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.672980 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:35:21.673072 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:35:21.673148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
I20260924 10:35:21.673312 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.668133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.2.193.129:38325)
0924 10:35:21.668353 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.668357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.668370 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:35:21.668418 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.668420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.668461 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.668822 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:21.668916 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.670696 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.672321 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.672744 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.672819 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.672866 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.673079 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.673133 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260924 10:35:21.674060 10289 tablet_bootstrap.cc:492] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: Bootstrap starting.
I20260924 10:35:21.674962 10289 tablet_bootstrap.cc:654] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:21.675292 9858 tablet_service.cc:1467] Processing CreateTablet for tablet c536933936f84b9bbde360e081cdbfd7 (DEFAULT_TABLE table=test-table [id=c1b5ff10ba8341a2a01e15dfe493b301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:21.675464 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.666988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.2.193.130:36309)
0924 10:35:21.667200 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.667206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.667281 (+ 75us) server_negotiation.cc:407] Connection header received
0924 10:35:21.667402 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.667405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.667452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.667536 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:21.667777 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.668509 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.669662 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.670561 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.675137 (+ 4576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.675207 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.675253 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.675319 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260924 10:35:21.675560 9858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c536933936f84b9bbde360e081cdbfd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:21.675926 9995 tablet_service.cc:1467] Processing CreateTablet for tablet c536933936f84b9bbde360e081cdbfd7 (DEFAULT_TABLE table=test-table [id=c1b5ff10ba8341a2a01e15dfe493b301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:21.676185 9995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c536933936f84b9bbde360e081cdbfd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:21.677865 10291 tablet_bootstrap.cc:492] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e: Bootstrap starting.
I20260924 10:35:21.678052 10289 tablet_bootstrap.cc:492] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: No bootstrap required, opened a new log
I20260924 10:35:21.678138 10289 ts_tablet_manager.cc:1397] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:35:21.678423 10292 tablet_bootstrap.cc:492] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c: Bootstrap starting.
I20260924 10:35:21.678649 10291 tablet_bootstrap.cc:654] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:21.678578 10289 raft_consensus.cc:348] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.678709 10289 raft_consensus.cc:374] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:21.678742 10289 raft_consensus.cc:729] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b504c9120404b93a9f5752c140b8013, State: Initialized, Role: FOLLOWER
I20260924 10:35:21.678879 10289 consensus_queue.cc:260] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.679230 10289 ts_tablet_manager.cc:1428] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:21.680620 10292 tablet_bootstrap.cc:654] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:21.681754 10197 heartbeater.cc:503] Master 127.2.193.188:45491 was elected leader, sending a full tablet report...
I20260924 10:35:21.681958 10291 tablet_bootstrap.cc:492] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e: No bootstrap required, opened a new log
I20260924 10:35:21.682041 10291 ts_tablet_manager.cc:1397] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:35:21.682184 10292 tablet_bootstrap.cc:492] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c: No bootstrap required, opened a new log
I20260924 10:35:21.682261 10292 ts_tablet_manager.cc:1397] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:35:21.682548 10291 raft_consensus.cc:348] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.682665 10291 raft_consensus.cc:374] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:21.682716 10291 raft_consensus.cc:729] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8bb19345504f579f0b7e49c8e4f96e, State: Initialized, Role: FOLLOWER
I20260924 10:35:21.682685 10292 raft_consensus.cc:348] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.682793 10292 raft_consensus.cc:374] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:21.682833 10292 raft_consensus.cc:729] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba188de6cd214f1cad09f370758fb39c, State: Initialized, Role: FOLLOWER
I20260924 10:35:21.682838 10291 consensus_queue.cc:260] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.682978 10292 consensus_queue.cc:260] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.683199 10291 ts_tablet_manager.cc:1428] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 10:35:21.683288 10292 ts_tablet_manager.cc:1428] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:21.684753 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.684350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.129:52289)
0924 10:35:21.684576 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.684639 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260924 10:35:21.685513 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.681840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.131:43537)
0924 10:35:21.685333 (+ 3493us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.685412 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3416,"thread_start_us":94,"threads_started":1}
I20260924 10:35:21.685729 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.685456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.129:41103)
0924 10:35:21.685601 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.685658 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:35:21.685833 9924 heartbeater.cc:503] Master 127.2.193.188:45491 was elected leader, sending a full tablet report...
I20260924 10:35:21.686302 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.684261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:51629)
0924 10:35:21.686132 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.686201 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1813,"thread_start_us":103,"threads_started":1}
I20260924 10:35:21.687014 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.686770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:57739)
0924 10:35:21.686891 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.686947 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:35:21.687252 10061 heartbeater.cc:503] Master 127.2.193.188:45491 was elected leader, sending a full tablet report...
I20260924 10:35:21.741875 10298 raft_consensus.cc:482] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:21.742044 10298 raft_consensus.cc:504] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.743288 10298 leader_election.cc:288] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b504c9120404b93a9f5752c140b8013 (127.2.193.131:45253), ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:21.747099 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.743600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:47643 (local address 127.2.193.131:45253)
0924 10:35:21.743741 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.743745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.743763 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:35:21.743829 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.743831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.743884 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.743992 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:21.744293 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.744892 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.746091 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.746508 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.746601 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.746681 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.746875 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.746959 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:35:21.747102 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.743577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38325 (local address 127.2.193.130:36771)
0924 10:35:21.743912 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.743927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.744024 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.744275 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:21.744278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:21.744294 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:21.744389 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.744394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.745286 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.745288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.746005 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.746013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.746674 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.746678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.746804 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.746833 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:35:21.746907 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:35:21.746975 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
I20260924 10:35:21.747110 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.743763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:36771 (local address 127.2.193.129:38325)
0924 10:35:21.743886 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.743889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.744009 (+ 120us) server_negotiation.cc:407] Connection header received
0924 10:35:21.744104 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.744107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.744158 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.744240 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:21.744531 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.745160 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.746145 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.746535 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.746671 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.746723 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.746934 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.747003 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:35:21.747104 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.743506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:45253 (local address 127.2.193.130:47643)
0924 10:35:21.743620 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.743639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.743744 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.744030 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:21.744035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:21.744055 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:21.744179 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.744189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.744995 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.744999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.745963 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:21.745972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.746631 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.746635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:21.746733 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.746762 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:35:21.746864 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:35:21.746961 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 10:35:21.747642 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c536933936f84b9bbde360e081cdbfd7" candidate_uuid: "ba188de6cd214f1cad09f370758fb39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" is_pre_election: true
I20260924 10:35:21.747800 9878 raft_consensus.cc:2454] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba188de6cd214f1cad09f370758fb39c in term 0.
I20260924 10:35:21.747884 10151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c536933936f84b9bbde360e081cdbfd7" candidate_uuid: "ba188de6cd214f1cad09f370758fb39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b504c9120404b93a9f5752c140b8013" is_pre_election: true
I20260924 10:35:21.748126 10151 raft_consensus.cc:2454] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba188de6cd214f1cad09f370758fb39c in term 0.
I20260924 10:35:21.748188 9951 leader_election.cc:302] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae8bb19345504f579f0b7e49c8e4f96e, ba188de6cd214f1cad09f370758fb39c; no voters:
I20260924 10:35:21.748485 10298 raft_consensus.cc:2781] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:21.748551 10298 raft_consensus.cc:482] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:21.748588 10298 raft_consensus.cc:3037] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:21.749606 10298 raft_consensus.cc:504] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.749997 10298 leader_election.cc:288] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [CANDIDATE]: Term 1 election: Requested vote from peers 7b504c9120404b93a9f5752c140b8013 (127.2.193.131:45253), ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325)
I20260924 10:35:21.750330 10151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c536933936f84b9bbde360e081cdbfd7" candidate_uuid: "ba188de6cd214f1cad09f370758fb39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b504c9120404b93a9f5752c140b8013"
I20260924 10:35:21.750339 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c536933936f84b9bbde360e081cdbfd7" candidate_uuid: "ba188de6cd214f1cad09f370758fb39c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8bb19345504f579f0b7e49c8e4f96e"
I20260924 10:35:21.750453 10151 raft_consensus.cc:3037] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:21.750505 9878 raft_consensus.cc:3037] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:21.751420 10151 raft_consensus.cc:2454] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba188de6cd214f1cad09f370758fb39c in term 1.
I20260924 10:35:21.751462 9878 raft_consensus.cc:2454] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba188de6cd214f1cad09f370758fb39c in term 1.
I20260924 10:35:21.751708 9948 leader_election.cc:302] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b504c9120404b93a9f5752c140b8013, ba188de6cd214f1cad09f370758fb39c; no voters:
I20260924 10:35:21.751915 10298 raft_consensus.cc:2781] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:21.752122 10298 raft_consensus.cc:686] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 1 LEADER]: Becoming Leader. State: Replica: ba188de6cd214f1cad09f370758fb39c, State: Running, Role: LEADER
I20260924 10:35:21.752399 10298 consensus_queue.cc:237] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } }
I20260924 10:35:21.752877 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.752582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:36549)
0924 10:35:21.752733 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.752792 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 10:35:21.755025 9743 catalog_manager.cc:5495] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c reported cstate change: term changed from 0 to 1, leader changed from <none> to ba188de6cd214f1cad09f370758fb39c (127.2.193.130). New cstate: current_term: 1 leader_uuid: "ba188de6cd214f1cad09f370758fb39c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba188de6cd214f1cad09f370758fb39c" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 36309 } health_report { overall_health: HEALTHY } } }
I20260924 10:35:21.778452 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.774800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.2.193.130:36309)
0924 10:35:21.774948 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:21.774953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:21.774968 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:35:21.775132 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:21.775135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:21.775195 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:21.775316 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:21.775565 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.776071 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.776939 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:21.777160 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:21.777298 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:21.777579 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:21.778038 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:21.778170 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:21.778233 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 10:35:21.778291 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:35:21.781126 9878 raft_consensus.cc:1261] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Refusing update from remote peer ba188de6cd214f1cad09f370758fb39c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:21.781455 10151 raft_consensus.cc:1261] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 1 FOLLOWER]: Refusing update from remote peer ba188de6cd214f1cad09f370758fb39c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:21.781756 10302 consensus_queue.cc:1035] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8bb19345504f579f0b7e49c8e4f96e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:21.782033 10298 consensus_queue.cc:1035] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b504c9120404b93a9f5752c140b8013" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:21.783990 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.783240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.129:54837)
0924 10:35:21.783820 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.783888 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507}
I20260924 10:35:21.784410 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.784130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.131:53797)
0924 10:35:21.784274 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.784335 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 10:35:21.786268 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.786004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:55655)
0924 10:35:21.786141 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.786195 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 10:35:21.788228 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.788019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.129:49821)
0924 10:35:21.788110 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.788161 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 10:35:21.792738 10307 mvcc.cc:205] Tried to move back new op lower bound from 7332848114810679296 to 7332848114701148160. Current Snapshot: MvccSnapshot[applied={T|T < 7332848114810679296}]
I20260924 10:35:21.793658 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.793412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.130:34535)
0924 10:35:21.793527 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.793584 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:35:21.794126 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.792810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40919 (local address 127.2.193.131:48975)
0924 10:35:21.793994 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.794050 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1108}
I20260924 10:35:21.797241 9745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:21.797439 9745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:21.800894 9745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41eb1f74cba6410d98bd5768c7626cfb: Sending DeleteTablet for 3 replicas of tablet c536933936f84b9bbde360e081cdbfd7
I20260924 10:35:21.801818 9858 tablet_service.cc:1514] Processing DeleteTablet for tablet c536933936f84b9bbde360e081cdbfd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:21 UTC) from {username='oryx', principal='oryx/127.2.193.188@KRBTEST.COM'} at 127.0.0.1:57844
I20260924 10:35:21.802032 10131 tablet_service.cc:1514] Processing DeleteTablet for tablet c536933936f84b9bbde360e081cdbfd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:21 UTC) from {username='oryx', principal='oryx/127.2.193.188@KRBTEST.COM'} at 127.0.0.1:38794
I20260924 10:35:21.802223 10315 tablet_replica.cc:330] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e: stopping tablet replica
I20260924 10:35:21.802358 10315 raft_consensus.cc:2229] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:21.802479 10316 tablet_replica.cc:330] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: stopping tablet replica
I20260924 10:35:21.802505 10315 raft_consensus.cc:2258] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:21.802685 10316 raft_consensus.cc:2229] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:21.802855 10316 raft_consensus.cc:2258] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:21.803143 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 9706
I20260924 10:35:21.803308 10315 ts_tablet_manager.cc:1905] T c536933936f84b9bbde360e081cdbfd7 P ae8bb19345504f579f0b7e49c8e4f96e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:21.806231 9995 tablet_service.cc:1514] Processing DeleteTablet for tablet c536933936f84b9bbde360e081cdbfd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:21 UTC) from {username='oryx', principal='oryx/127.2.193.188@KRBTEST.COM'} at 127.0.0.1:55220
I20260924 10:35:21.806883 10316 ts_tablet_manager.cc:1905] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:21.810431 10316 ts_tablet_manager.cc:1918] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:21.810520 10316 log.cc:1196] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/c536933936f84b9bbde360e081cdbfd7
I20260924 10:35:21.810786 10317 tablet_replica.cc:330] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c: stopping tablet replica
I20260924 10:35:21.810827 10316 ts_tablet_manager.cc:1939] T c536933936f84b9bbde360e081cdbfd7 P 7b504c9120404b93a9f5752c140b8013: Deleting consensus metadata
I20260924 10:35:21.811000 10317 raft_consensus.cc:2229] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:21.811358 10317 raft_consensus.cc:2258] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:21.811697 9729 catalog_manager.cc:4841] TS 7b504c9120404b93a9f5752c140b8013 (127.2.193.131:45253): tablet c536933936f84b9bbde360e081cdbfd7 (table test-table [id=c1b5ff10ba8341a2a01e15dfe493b301]) successfully deleted
W20260924 10:35:21.812315 10287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:21.811439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38325 (local address 127.0.0.1:57854)
0924 10:35:21.811609 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.811627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:21.811709 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:21.812208 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38325: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 10:35:21.812366 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 9932
I20260924 10:35:21.812466 10317 ts_tablet_manager.cc:1905] T c536933936f84b9bbde360e081cdbfd7 P ba188de6cd214f1cad09f370758fb39c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:35:21.812492 9732 catalog_manager.cc:4584] TS ae8bb19345504f579f0b7e49c8e4f96e (127.2.193.129:38325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c536933936f84b9bbde360e081cdbfd7: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38325: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:35:21.818281 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 10068
I20260924 10:35:21.819176 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:21.818390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:36309 (local address 127.0.0.1:55230)
0924 10:35:21.819006 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:21.819071 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523}
I20260924 10:35:21.824962 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 9424
I20260924 10:35:21.830623 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 9533
2026-09-24T10:35:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5360 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:21 dist-test-slave-8wc7 krb5kdc[10320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:21 dist-test-slave-8wc7 krb5kdc[10320](info): set up 2 sockets
Sep 24 10:35:21 dist-test-slave-8wc7 krb5kdc[10320](info): commencing operation
krb5kdc: starting...
W20260924 10:35:22.873868 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:22 dist-test-slave-8wc7 krb5kdc[10320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab.
Entry for principal oryx/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab.
I20260924 10:35:23.000811 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40939
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab
--principal=oryx/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:23.113233 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:23.113548 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:23.113591 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:23.117877 10333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:23.117945 10333 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:35:23.117964 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:23.117982 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:23.118034 10333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:23.118073 10333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:23.123585 10333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.190.keytab
--principal=oryx/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:23.124776 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:23.125895 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:23.134873 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:23.134864 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:23.134903 10333 server_base.cc:1034] running on GCE node
W20260924 10:35:23.134864 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:23.135713 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:23.136950 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:23.138128 10333 hybrid_clock.cc:648] HybridClock initialized: now 1790246123138090 us; error 53 us; skew 500 ppm
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:23.141074 10333 init.cc:377] Logged in from keytab as oryx/127.2.193.190@KRBTEST.COM (short username oryx)
I20260924 10:35:23.142048 10333 webserver.cc:460] Webserver started at http://127.2.193.190:42813/ using document root <none> and password file <none>
I20260924 10:35:23.142719 10333 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:23.142797 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:23.142982 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:23.145126 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "172536804a9b4669bd5d016fee4025dc"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "1058dad63ee375b9f3f2e4215f9f53e7"
server_key_iv: "481a0b384e5f8f325273189802167a7f"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.145753 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "172536804a9b4669bd5d016fee4025dc"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "1058dad63ee375b9f3f2e4215f9f53e7"
server_key_iv: "481a0b384e5f8f325273189802167a7f"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.149753 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:35:23.152257 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.153687 10333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:23.153837 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "172536804a9b4669bd5d016fee4025dc"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "1058dad63ee375b9f3f2e4215f9f53e7"
server_key_iv: "481a0b384e5f8f325273189802167a7f"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.153965 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:23.190476 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:23.194231 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:23.194478 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:23.202859 10333 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:40939
I20260924 10:35:23.202878 10410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:40939 every 8 connection(s)
I20260924 10:35:23.203900 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:23.206843 10411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:23.207657 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 10333
I20260924 10:35:23.207814 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:23.208106 2822 external_mini_cluster.cc:1428] Setting key 3a72f0fc14c95f93d9d8ce0b75b579cd
I20260924 10:35:23.214048 10411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Bootstrap starting.
I20260924 10:35:23.216732 10411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:23.217607 10411 log.cc:824] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:23.219911 10411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: No bootstrap required, opened a new log
I20260924 10:35:23.223591 10411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172536804a9b4669bd5d016fee4025dc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40939 } }
I20260924 10:35:23.223858 10411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:23.223922 10411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172536804a9b4669bd5d016fee4025dc, State: Initialized, Role: FOLLOWER
I20260924 10:35:23.224411 10411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172536804a9b4669bd5d016fee4025dc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40939 } }
I20260924 10:35:23.224538 10411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:23.224625 10411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:23.224745 10411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:23.224763 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.209638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.2.193.190:40939)
0924 10:35:23.210108 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:23.210127 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:23.210175 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:35:23.210997 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:23.211021 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:23.211473 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:23.211874 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:23.212579 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.214099 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.215016 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.215746 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.215916 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.219021 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:23.219026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:23.219041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:23.219068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:23.221504 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.222061 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.222067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.222072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.222196 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.222517 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.222519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.222521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.222943 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:23.223102 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:23.223426 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 10:35:23.223774 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20260924 10:35:23.225827 10411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172536804a9b4669bd5d016fee4025dc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40939 } }
I20260924 10:35:23.226194 10411 leader_election.cc:302] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 172536804a9b4669bd5d016fee4025dc; no voters:
I20260924 10:35:23.226519 10411 leader_election.cc:288] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:23.226616 10416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:23.226820 10416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [term 1 LEADER]: Becoming Leader. State: Replica: 172536804a9b4669bd5d016fee4025dc, State: Running, Role: LEADER
I20260924 10:35:23.227435 10416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172536804a9b4669bd5d016fee4025dc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40939 } }
I20260924 10:35:23.227826 10411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:23.228579 10416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 172536804a9b4669bd5d016fee4025dc. Latest consensus state: current_term: 1 leader_uuid: "172536804a9b4669bd5d016fee4025dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172536804a9b4669bd5d016fee4025dc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40939 } } }
I20260924 10:35:23.228694 10416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:23.229406 10417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "172536804a9b4669bd5d016fee4025dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172536804a9b4669bd5d016fee4025dc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40939 } } }
I20260924 10:35:23.229493 10425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:23.229517 10417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:23.232977 10425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:23.240060 10425 catalog_manager.cc:1347] Generated new cluster ID: bbe645bd44114a67973ffc0005811c9b
I20260924 10:35:23.240160 10425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:23.254307 10425 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:23.255431 10425 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:23.265566 10425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Generated new TSK 0
I20260924 10:35:23.266360 10425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab.
Entry for principal oryx/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab.
I20260924 10:35:23.300729 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40939
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab
--principal=oryx/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:23.435678 10437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:23.435990 10437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:23.436038 10437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:23.440330 10437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:23.440410 10437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:23.440452 10437 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:35:23.440487 10437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:23.446260 10437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab
--principal=oryx/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:23.447474 10437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:23.448503 10437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:23.458559 10443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:23.458613 10442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:23.458827 10437 server_base.cc:1034] running on GCE node
W20260924 10:35:23.459034 10445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:23.459491 10437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:23.460222 10437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:23.461411 10437 hybrid_clock.cc:648] HybridClock initialized: now 1790246123461391 us; error 40 us; skew 500 ppm
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:23.464217 10437 init.cc:377] Logged in from keytab as oryx/127.2.193.129@KRBTEST.COM (short username oryx)
I20260924 10:35:23.465382 10437 webserver.cc:460] Webserver started at http://127.2.193.129:43319/ using document root <none> and password file <none>
I20260924 10:35:23.465998 10437 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:23.466094 10437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:23.466327 10437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:23.468520 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "d1bc5048aa7f4876b9614b1f2b967fa3"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "4da6a88379be7c1f8a9bf23aae4fc9f5"
server_key_iv: "f601dd5ea33d1a146a6ba70e4b8cae2b"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.469133 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "d1bc5048aa7f4876b9614b1f2b967fa3"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "4da6a88379be7c1f8a9bf23aae4fc9f5"
server_key_iv: "f601dd5ea33d1a146a6ba70e4b8cae2b"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.473462 10437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:23.476392 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.477982 10437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:23.478142 10437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "d1bc5048aa7f4876b9614b1f2b967fa3"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "4da6a88379be7c1f8a9bf23aae4fc9f5"
server_key_iv: "f601dd5ea33d1a146a6ba70e4b8cae2b"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.478278 10437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:23.494637 10437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:23.497994 10437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:23.498215 10437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:23.499794 10437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:23.499887 10437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.499961 10437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:23.500010 10437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:23.511210 10437 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:43919
I20260924 10:35:23.511276 10567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:43919 every 8 connection(s)
I20260924 10:35:23.512303 10437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:23.514884 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.500624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:49713 (local address 127.2.193.190:40939)
0924 10:35:23.500757 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:23.500760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:23.501959 (+ 1199us) server_negotiation.cc:407] Connection header received
0924 10:35:23.502971 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:23.502976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:23.503038 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:23.503104 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:23.504685 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.505624 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.507001 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.507474 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.507615 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.510923 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:23.510927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:23.510932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:23.510963 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:23.512607 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.513323 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.513326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.513328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.513417 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.513773 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.513777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.513779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.513975 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:23.514086 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:23.514576 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 10:35:23.514706 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:35:23.515684 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.501234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40939 (local address 127.2.193.129:49713)
0924 10:35:23.501776 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:23.501835 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:23.502700 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:23.503544 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:23.503552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:23.503909 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:23.504505 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.504529 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.505766 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.505770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.506890 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.506899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.507666 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.507672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.507872 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.509129 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:23.509150 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:23.510735 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:23.512739 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.512744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.512780 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.513139 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.513517 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.513522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.513524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.513668 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.514081 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:23.514088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:23.514337 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 10:35:23.514521 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:35:23.514860 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":144,"threads_started":1}
I20260924 10:35:23.516942 10568 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40939
I20260924 10:35:23.517100 10568 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:23.517612 10568 heartbeater.cc:511] Master 127.2.193.190:40939 requested a full tablet report, sending...
I20260924 10:35:23.518844 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 10437
I20260924 10:35:23.518996 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:23.519285 10364 ts_manager.cc:194] Registered new tserver with Master: d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919)
I20260924 10:35:23.519366 2822 external_mini_cluster.cc:1428] Setting key 678c82a953945635a0b1d8108465e3df
I20260924 10:35:23.520843 10364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:49713
I20260924 10:35:23.527447 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.519333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40939 (local address 127.2.193.129:49661)
0924 10:35:23.519643 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:23.519657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:23.519759 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:23.519997 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:23.520000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:23.520205 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:23.520313 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.520319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.521316 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.521320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.522147 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.522156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.522775 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.522779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.522868 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.523455 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:23.523469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:23.523824 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:23.525872 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.525875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.525876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.526072 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.526401 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.526403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.526405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.526480 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.526952 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:23.526954 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:23.527027 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 10:35:23.527102 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:35:23.527209 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260924 10:35:23.527522 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.519384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:49661 (local address 127.2.193.190:40939)
0924 10:35:23.519493 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:23.519497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:23.519747 (+ 250us) server_negotiation.cc:407] Connection header received
0924 10:35:23.519843 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:23.519846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:23.519899 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:23.519981 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:23.520427 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.521163 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.522264 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.522645 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.522766 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.523965 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:23.523971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:23.523977 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:23.524008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:23.525732 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.526179 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.526187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.526190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.526301 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.526575 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.526581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.526584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.526850 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:23.526959 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:23.527196 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 10:35:23.527344 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":43}
WARNING: no policy specified for oryx/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab.
Entry for principal oryx/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab.
I20260924 10:35:23.549515 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40939
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab
--principal=oryx/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:23.662788 10577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:23.663079 10577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:23.663125 10577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:23.667469 10577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:23.667541 10577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:23.667582 10577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:35:23.667646 10577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:23.673231 10577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab
--principal=oryx/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:23.674396 10577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:23.675483 10577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:23.685562 10585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:23.685624 10582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:23.685578 10577 server_base.cc:1034] running on GCE node
W20260924 10:35:23.685586 10583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:23.686239 10577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:23.687017 10577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:23.688190 10577 hybrid_clock.cc:648] HybridClock initialized: now 1790246123688168 us; error 34 us; skew 500 ppm
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:23.691082 10577 init.cc:377] Logged in from keytab as oryx/127.2.193.130@KRBTEST.COM (short username oryx)
I20260924 10:35:23.692014 10577 webserver.cc:460] Webserver started at http://127.2.193.130:43903/ using document root <none> and password file <none>
I20260924 10:35:23.692561 10577 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:23.692623 10577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:23.692791 10577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:23.694967 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "49e2280a31438f9c04028d305fb2a1a2"
server_key_iv: "3352e8ae693f8d3843469e0d27c54216"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.695567 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "49e2280a31438f9c04028d305fb2a1a2"
server_key_iv: "3352e8ae693f8d3843469e0d27c54216"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.699427 10577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 10:35:23.702317 10592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.703759 10577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:23.703918 10577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "49e2280a31438f9c04028d305fb2a1a2"
server_key_iv: "3352e8ae693f8d3843469e0d27c54216"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.704042 10577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:23.731073 10577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:23.734472 10577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:23.734691 10577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:23.736292 10577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:23.736353 10577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.736423 10577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:23.736454 10577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.747411 10577 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:45741
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:23.748538 10577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:23.748543 10707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:45741 every 8 connection(s)
I20260924 10:35:23.752310 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.737280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:39553 (local address 127.2.193.190:40939)
0924 10:35:23.737406 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:23.737410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:23.738625 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 10:35:23.739769 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:23.739772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:23.739856 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:23.739957 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:23.741799 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.742668 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.744679 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.744923 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.745020 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.748431 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:23.748435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:23.748438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:23.748474 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:23.750129 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.750679 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.750682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.750684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.750781 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.751132 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.751136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.751137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.751368 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:23.751456 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:23.752037 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 10:35:23.752150 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:35:23.753314 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.737924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40939 (local address 127.2.193.130:39553)
0924 10:35:23.738458 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:23.738502 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:23.739523 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:23.740215 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:23.740229 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:23.740742 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:23.741588 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.741616 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.742855 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.742859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.744472 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.744484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.745112 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.745120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.745402 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.746210 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:23.746233 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:23.748165 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:23.750241 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.750246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.750260 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.750545 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.750867 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.750869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.750870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.751007 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.751496 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:23.751502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:23.751764 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 10:35:23.751932 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 10:35:23.752204 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":142,"threads_started":1}
I20260924 10:35:23.754631 10708 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40939
I20260924 10:35:23.754786 10708 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:23.755364 10708 heartbeater.cc:511] Master 127.2.193.190:40939 requested a full tablet report, sending...
I20260924 10:35:23.756433 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 10577
I20260924 10:35:23.756553 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:23.756574 10365 ts_manager.cc:194] Registered new tserver with Master: e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741)
I20260924 10:35:23.756873 2822 external_mini_cluster.cc:1428] Setting key 63c802201b69a5b62e28a71a75988b88
I20260924 10:35:23.757395 10365 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:39553
WARNING: no policy specified for oryx/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:23.767477 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.757116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40939 (local address 127.2.193.130:52091)
0924 10:35:23.757475 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:23.757491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:23.757620 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:23.757918 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:23.757921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:23.759815 (+ 1894us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:23.759968 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.759979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.760804 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.760811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.761902 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.761916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.762434 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.762440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.762576 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.763137 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:23.763156 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:23.763655 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:23.765681 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.765684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.765687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.765956 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.766267 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.766272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.766275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.766377 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.766805 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:23.766808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:23.766919 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 10:35:23.767043 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:35:23.767208 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":121,"threads_started":1}
I20260924 10:35:23.767498 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.757161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:52091 (local address 127.2.193.190:40939)
0924 10:35:23.757326 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:23.757330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:23.757554 (+ 224us) server_negotiation.cc:407] Connection header received
0924 10:35:23.757736 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:23.757739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:23.757799 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:23.757882 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:23.760121 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.760650 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.762041 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.762288 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.762405 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.763810 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:23.763813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:23.763815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:23.763846 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:23.765545 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.766057 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.766060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.766062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.766163 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.766477 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.766479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.766480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.766699 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:23.766777 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:23.767268 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 10:35:23.767386 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab.
Entry for principal oryx/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab.
I20260924 10:35:23.786530 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40939
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab
--principal=oryx/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:23.902405 10717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:23.902697 10717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:23.902745 10717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:23.906808 10717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:23.906880 10717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:23.906922 10717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:35:23.906958 10717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:23.912528 10717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab
--principal=oryx/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.193.190:40939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:23.913666 10717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:23.914708 10717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:23.924830 10723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:23.924839 10725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:23.924840 10722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:23.925122 10717 server_base.cc:1034] running on GCE node
I20260924 10:35:23.925720 10717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:23.926415 10717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:23.927624 10717 hybrid_clock.cc:648] HybridClock initialized: now 1790246123927608 us; error 34 us; skew 500 ppm
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:23.930751 10717 init.cc:377] Logged in from keytab as oryx/127.2.193.131@KRBTEST.COM (short username oryx)
I20260924 10:35:23.931656 10717 webserver.cc:460] Webserver started at http://127.2.193.131:46679/ using document root <none> and password file <none>
I20260924 10:35:23.932253 10717 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:23.932317 10717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:23.932590 10717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:23.934810 10717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "121d57765a85492ab71c703f6870058e"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "91fb0cf03153858e80411b8f47bd089e"
server_key_iv: "f3c681e8e3af077088a48d733584dea3"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.935393 10717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "121d57765a85492ab71c703f6870058e"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "91fb0cf03153858e80411b8f47bd089e"
server_key_iv: "f3c681e8e3af077088a48d733584dea3"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.939482 10717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 10:35:23.942215 10732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.943653 10717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:23.943812 10717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "121d57765a85492ab71c703f6870058e"
format_stamp: "Formatted at 2026-09-24 10:35:23 on dist-test-slave-8wc7"
server_key: "91fb0cf03153858e80411b8f47bd089e"
server_key_iv: "f3c681e8e3af077088a48d733584dea3"
server_key_version: "encryptionkey@0"
I20260924 10:35:23.943949 10717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:23.962744 10717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:23.966500 10717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:23.966714 10717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:23.968250 10717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:23.968330 10717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.968408 10717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:23.968451 10717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:23.978683 10717 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:41669
I20260924 10:35:23.978773 10847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:41669 every 8 connection(s)
Sep 24 10:35:23 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:23.979739 10717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:23.983578 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 10717
I20260924 10:35:23.983716 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:23.983727 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.969163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:37225 (local address 127.2.193.190:40939)
0924 10:35:23.969342 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:23.969347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:23.970309 (+ 962us) server_negotiation.cc:407] Connection header received
0924 10:35:23.971512 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:23.971518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:23.971588 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:23.971701 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:23.973639 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.974550 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.976506 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.976915 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.977047 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.980093 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:23.980098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:23.980101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:23.980138 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:23.981734 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.982225 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.982228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.982230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.982308 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.982692 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.982694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.982695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.982872 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:23.982992 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:23.983448 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 10:35:23.983567 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:35:23.984036 2822 external_mini_cluster.cc:1428] Setting key bbd126da1b79afa4aa6b31a56d9722b4
I20260924 10:35:23.984876 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.969633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40939 (local address 127.2.193.131:37225)
0924 10:35:23.970103 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:23.970155 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:23.971213 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:23.972011 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:23.972020 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:23.972605 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:23.973439 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.973467 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.974721 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.974729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.976326 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.976340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.977115 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.977120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.977449 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.978135 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:23.978153 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:23.979883 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:23.981851 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.981856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.981867 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.982125 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.982450 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.982455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.982460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.982603 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.982955 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:23.982960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:23.983188 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 10:35:23.983393 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 10:35:23.983773 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":142,"threads_started":1}
I20260924 10:35:23.986240 10848 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40939
I20260924 10:35:23.986397 10848 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:23.986893 10848 heartbeater.cc:511] Master 127.2.193.190:40939 requested a full tablet report, sending...
I20260924 10:35:23.987849 10364 ts_manager.cc:194] Registered new tserver with Master: 121d57765a85492ab71c703f6870058e (127.2.193.131:41669)
I20260924 10:35:23.988507 10364 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:37225
I20260924 10:35:23.994951 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.988453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:43915 (local address 127.2.193.190:40939)
0924 10:35:23.988582 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:23.988586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:23.988841 (+ 255us) server_negotiation.cc:407] Connection header received
0924 10:35:23.988992 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:23.989016 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:23.989062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:23.989136 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:23.989588 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.990298 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.991095 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.991298 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.991376 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.992416 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:23.992419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:23.992421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:23.992436 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:23.993659 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.994033 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.994036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.994037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.994103 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:23.994321 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:23.994323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:23.994324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:23.994473 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:23.994543 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:23.994752 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 10:35:23.994839 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:35:23.994959 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:23.988339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40939 (local address 127.2.193.131:43915)
0924 10:35:23.988745 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:23.988759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:23.988884 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:23.989145 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:23.989147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:23.989377 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:23.989476 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.989482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.990420 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.990423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.990974 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:23.990982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:23.991392 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:23.991395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:23.991466 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:23.991970 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:23.991984 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:23.992308 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:23.993752 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.993754 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.993756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.993951 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.994178 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:23.994180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:23.994181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:23.994249 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:23.994547 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:23.994549 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:23.994620 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:35:23.994698 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:35:23.994804 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":102,"threads_started":1}
I20260924 10:35:23.997885 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246124, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:35:24.022760 10414 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:24.016129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.2.193.190:40939)
0924 10:35:24.016290 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.016294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.016309 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:35:24.016353 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.016356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.016396 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.016500 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:24.017324 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.017846 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.018853 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.019086 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.019193 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.022636 (+ 3443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60744: BlockingRecv error: recv got EOF from 127.0.0.1:60744 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246124, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:24.033423 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.024311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.2.193.190:40939)
0924 10:35:24.024447 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.024454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.024668 (+ 214us) server_negotiation.cc:407] Connection header received
0924 10:35:24.024830 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.024835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.024883 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.024971 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:24.025645 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.026164 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.026986 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.027207 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.027338 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.030033 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:24.030036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:24.030039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:24.030099 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:24.031910 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:24.032326 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:24.032329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:24.032331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:24.032405 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:24.032636 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:24.032638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:24.032639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:24.032819 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:24.032938 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.033122 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.033291 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:35:24.036459 10364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:24.039845 10364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:24.054833 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.048691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:45741 (local address 127.0.0.1:52992)
0924 10:35:24.049099 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.049162 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.049328 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.050609 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.050612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.050629 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:24.050721 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.050726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.052124 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.052127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.053320 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.053329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.054356 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.054359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.054493 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.054512 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.054567 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.054667 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":70,"thread_start_us":121,"threads_started":1}
I20260924 10:35:24.054885 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.049745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40726 (local address 127.2.193.129:43919)
0924 10:35:24.050001 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.050007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.050419 (+ 412us) server_negotiation.cc:407] Connection header received
0924 10:35:24.050493 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.050498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.050752 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.050916 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:24.051112 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.052187 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.053459 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.054110 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.054215 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.054375 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.054668 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.054762 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260924 10:35:24.054950 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.048868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.2.193.130:45741)
0924 10:35:24.050142 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.050156 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.050180 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:35:24.050251 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.050257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.050444 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.050640 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:24.050839 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.052003 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.053446 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.054249 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.054433 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.054612 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.054665 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.054759 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":124,"threads_started":1}
I20260924 10:35:24.055234 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.049084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41669 (local address 127.0.0.1:40160)
0924 10:35:24.049307 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.049321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.049420 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.050272 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.050278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.050299 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:24.050479 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.050486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.051767 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.051772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.053224 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.053234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.054318 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.054324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.054444 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.054465 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.054515 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.054673 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":58,"spinlock_wait_cycles":2944,"thread_start_us":595,"threads_started":1}
I20260924 10:35:24.055641 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.049507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:43919 (local address 127.0.0.1:40726)
0924 10:35:24.050238 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.050250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.050333 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.050894 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.050896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.050914 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:24.051001 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.051006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.052292 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.052297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.053357 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.053367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.054214 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.054217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.054330 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.054368 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.054488 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.055552 (+ 1064us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":918,"thread_start_us":98,"threads_started":1}
I20260924 10:35:24.056070 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.049423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.2.193.131:41669)
0924 10:35:24.049722 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.049734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.049757 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:35:24.049834 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.049839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.050068 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.050639 (+ 571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:24.050766 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.051650 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.053379 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.054173 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.054743 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.054896 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.055827 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.055939 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260924 10:35:24.058187 10642 tablet_service.cc:1467] Processing CreateTablet for tablet 1444b6a3df694ac8be4d22e2ecc83b85 (DEFAULT_TABLE table=test-table [id=b8dd3c822c494a1c897588462a638c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.058480 10782 tablet_service.cc:1467] Processing CreateTablet for tablet 1444b6a3df694ac8be4d22e2ecc83b85 (DEFAULT_TABLE table=test-table [id=b8dd3c822c494a1c897588462a638c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.058470 10502 tablet_service.cc:1467] Processing CreateTablet for tablet 1444b6a3df694ac8be4d22e2ecc83b85 (DEFAULT_TABLE table=test-table [id=b8dd3c822c494a1c897588462a638c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.059412 10642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1444b6a3df694ac8be4d22e2ecc83b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.059489 10502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1444b6a3df694ac8be4d22e2ecc83b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.059492 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1444b6a3df694ac8be4d22e2ecc83b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.066160 10876 tablet_bootstrap.cc:492] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Bootstrap starting.
I20260924 10:35:24.067306 10877 tablet_bootstrap.cc:492] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Bootstrap starting.
I20260924 10:35:24.067730 10878 tablet_bootstrap.cc:492] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Bootstrap starting.
I20260924 10:35:24.068483 10876 tablet_bootstrap.cc:654] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.069288 10876 log.cc:824] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:24.069415 10877 tablet_bootstrap.cc:654] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.069646 10878 tablet_bootstrap.cc:654] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.070242 10877 log.cc:824] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:24.070346 10878 log.cc:824] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:24.071491 10876 tablet_bootstrap.cc:492] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: No bootstrap required, opened a new log
I20260924 10:35:24.071686 10876 ts_tablet_manager.cc:1397] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:35:24.072139 10878 tablet_bootstrap.cc:492] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: No bootstrap required, opened a new log
I20260924 10:35:24.072275 10878 ts_tablet_manager.cc:1397] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:35:24.072360 10877 tablet_bootstrap.cc:492] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: No bootstrap required, opened a new log
I20260924 10:35:24.072541 10877 ts_tablet_manager.cc:1397] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:35:24.075232 10878 raft_consensus.cc:348] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.075502 10878 raft_consensus.cc:374] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.075562 10878 raft_consensus.cc:729] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d57765a85492ab71c703f6870058e, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.076014 10878 consensus_queue.cc:260] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.075906 10876 raft_consensus.cc:348] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.076166 10876 raft_consensus.cc:374] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.076228 10876 raft_consensus.cc:729] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c472bc605b4a7ea9b0e6cd974de1c1, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.076682 10848 heartbeater.cc:503] Master 127.2.193.190:40939 was elected leader, sending a full tablet report...
I20260924 10:35:24.076669 10876 consensus_queue.cc:260] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.077075 10878 ts_tablet_manager.cc:1428] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260924 10:35:24.077011 10877 raft_consensus.cc:348] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.077260 10877 raft_consensus.cc:374] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.077306 10877 raft_consensus.cc:729] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bc5048aa7f4876b9614b1f2b967fa3, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.077644 10876 ts_tablet_manager.cc:1428] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:35:24.077685 10877 consensus_queue.cc:260] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.077975 10708 heartbeater.cc:503] Master 127.2.193.190:40939 was elected leader, sending a full tablet report...
I20260924 10:35:24.078320 10568 heartbeater.cc:503] Master 127.2.193.190:40939 was elected leader, sending a full tablet report...
I20260924 10:35:24.078637 10877 ts_tablet_manager.cc:1428] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:35:24.166707 10882 raft_consensus.cc:482] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:24.166985 10882 raft_consensus.cc:504] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.168111 10882 leader_election.cc:288] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741), d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919)
I20260924 10:35:24.172751 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.168434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:45741 (local address 127.2.193.131:59943)
0924 10:35:24.168578 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.168596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.168697 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.169079 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.169082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.169105 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:24.169265 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.169272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.170265 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.170268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.171601 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.171611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.172416 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.172419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.172507 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.172527 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.172587 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.172646 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:35:24.172818 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.168537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:59943 (local address 127.2.193.130:45741)
0924 10:35:24.168658 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.168661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.168698 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:35:24.168800 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.168803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.168857 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.168953 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:24.169398 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.170157 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.171748 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.172250 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.172425 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.172478 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.172620 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.172691 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:35:24.173430 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.168733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:43919 (local address 127.2.193.131:57323)
0924 10:35:24.169069 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.169082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.169165 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.169399 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.169401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.169417 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:24.169528 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.169533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.170602 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.170605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.172022 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.172029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.173057 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.173061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.173161 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.173213 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.173276 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.173330 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":169,"threads_started":1}
I20260924 10:35:24.173511 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.168895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:57323 (local address 127.2.193.129:43919)
0924 10:35:24.169017 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.169021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.169162 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:35:24.169258 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.169265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.169306 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.169393 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:24.169822 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.170488 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.172157 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.172782 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.172921 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.172978 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.173322 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.173402 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:35:24.173506 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444b6a3df694ac8be4d22e2ecc83b85" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" is_pre_election: true
I20260924 10:35:24.173813 10662 raft_consensus.cc:2454] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:24.174319 10733 leader_election.cc:302] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, e6c472bc605b4a7ea9b0e6cd974de1c1; no voters:
I20260924 10:35:24.174376 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444b6a3df694ac8be4d22e2ecc83b85" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" is_pre_election: true
I20260924 10:35:24.174623 10882 raft_consensus.cc:2781] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:24.174710 10882 raft_consensus.cc:482] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:24.174679 10522 raft_consensus.cc:2454] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:24.174742 10882 raft_consensus.cc:3037] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.175874 10882 raft_consensus.cc:504] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.176226 10882 leader_election.cc:288] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Requested vote from peers e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741), d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919)
I20260924 10:35:24.176643 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444b6a3df694ac8be4d22e2ecc83b85" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
I20260924 10:35:24.176765 10662 raft_consensus.cc:3037] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.176790 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1444b6a3df694ac8be4d22e2ecc83b85" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3"
I20260924 10:35:24.176947 10522 raft_consensus.cc:3037] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.177886 10662 raft_consensus.cc:2454] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:24.178085 10522 raft_consensus.cc:2454] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:24.178208 10733 leader_election.cc:302] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, e6c472bc605b4a7ea9b0e6cd974de1c1; no voters:
I20260924 10:35:24.178475 10882 raft_consensus.cc:2781] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:24.178659 10882 raft_consensus.cc:686] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 1 LEADER]: Becoming Leader. State: Replica: 121d57765a85492ab71c703f6870058e, State: Running, Role: LEADER
I20260924 10:35:24.178982 10882 consensus_queue.cc:237] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.182271 10363 catalog_manager.cc:5495] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e reported cstate change: term changed from 0 to 1, leader changed from <none> to 121d57765a85492ab71c703f6870058e (127.2.193.131). New cstate: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } health_report { overall_health: HEALTHY } } }
I20260924 10:35:24.209578 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.204997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.2.193.131:41669)
0924 10:35:24.205130 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.205135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.205151 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:35:24.205314 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.205317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.205377 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.205494 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:24.205727 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.206523 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.207714 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.208041 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.208148 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.208585 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:24.209114 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:24.209240 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.209373 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.209435 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:35:24.216219 10662 raft_consensus.cc:1261] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.216288 10522 raft_consensus.cc:1261] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.217010 10882 consensus_queue.cc:1035] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.217224 10886 consensus_queue.cc:1035] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.224262 10889 mvcc.cc:205] Tried to move back new op lower bound from 7332848124781764608 to 7332848124640055296. Current Snapshot: MvccSnapshot[applied={T|T < 7332848124781764608}]
I20260924 10:35:24.225841 10891 mvcc.cc:205] Tried to move back new op lower bound from 7332848124781764608 to 7332848124640055296. Current Snapshot: MvccSnapshot[applied={T|T < 7332848124781764608}]
I20260924 10:35:24.226444 10890 mvcc.cc:205] Tried to move back new op lower bound from 7332848124781764608 to 7332848124640055296. Current Snapshot: MvccSnapshot[applied={T|T < 7332848124781764608}]
I20260924 10:35:24.233626 10364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:24.233784 10364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:24.236356 10364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Sending DeleteTablet for 3 replicas of tablet 1444b6a3df694ac8be4d22e2ecc83b85
I20260924 10:35:24.237417 10782 tablet_service.cc:1514] Processing DeleteTablet for tablet 1444b6a3df694ac8be4d22e2ecc83b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40160
I20260924 10:35:24.237728 10502 tablet_service.cc:1514] Processing DeleteTablet for tablet 1444b6a3df694ac8be4d22e2ecc83b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40726
I20260924 10:35:24.237968 10642 tablet_service.cc:1514] Processing DeleteTablet for tablet 1444b6a3df694ac8be4d22e2ecc83b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52992
I20260924 10:35:24.238224 10899 tablet_replica.cc:330] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: stopping tablet replica
I20260924 10:35:24.238417 10898 tablet_replica.cc:330] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: stopping tablet replica
I20260924 10:35:24.238564 10899 raft_consensus.cc:2229] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:24.238709 10898 raft_consensus.cc:2229] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:24.238883 10899 raft_consensus.cc:2258] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.239120 10898 raft_consensus.cc:2258] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.239552 10900 tablet_replica.cc:330] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: stopping tablet replica
I20260924 10:35:24.239919 10900 raft_consensus.cc:2229] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:24.240003 10364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:35:24.240437 10900 raft_consensus.cc:2258] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.240435 10899 ts_tablet_manager.cc:1905] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:35:24.240474 10364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:24.243212 10898 ts_tablet_manager.cc:1905] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.243480 10899 ts_tablet_manager.cc:1918] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:24.243571 10899 log.cc:1196] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/1444b6a3df694ac8be4d22e2ecc83b85
I20260924 10:35:24.243887 10899 ts_tablet_manager.cc:1939] T 1444b6a3df694ac8be4d22e2ecc83b85 P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting consensus metadata
I20260924 10:35:24.245136 10349 catalog_manager.cc:4841] TS d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919): tablet 1444b6a3df694ac8be4d22e2ecc83b85 (table test-table [id=b8dd3c822c494a1c897588462a638c85]) successfully deleted
I20260924 10:35:24.247463 10900 ts_tablet_manager.cc:1905] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.249827 10642 tablet_service.cc:1467] Processing CreateTablet for tablet b5cee6e894f342f4b84a2a06f6d6659a (DEFAULT_TABLE table=test-table [id=a9ddb7932fa74efa9811e15200181806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.249929 10502 tablet_service.cc:1467] Processing CreateTablet for tablet b5cee6e894f342f4b84a2a06f6d6659a (DEFAULT_TABLE table=test-table [id=a9ddb7932fa74efa9811e15200181806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.250097 10642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cee6e894f342f4b84a2a06f6d6659a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.250186 10502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cee6e894f342f4b84a2a06f6d6659a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.250654 10900 ts_tablet_manager.cc:1918] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:24.250825 10900 log.cc:1196] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/1444b6a3df694ac8be4d22e2ecc83b85
I20260924 10:35:24.251210 10900 ts_tablet_manager.cc:1939] T 1444b6a3df694ac8be4d22e2ecc83b85 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting consensus metadata
I20260924 10:35:24.252198 10876 tablet_bootstrap.cc:492] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Bootstrap starting.
I20260924 10:35:24.252499 10877 tablet_bootstrap.cc:492] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Bootstrap starting.
I20260924 10:35:24.253155 10876 tablet_bootstrap.cc:654] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.253525 10782 tablet_service.cc:1467] Processing CreateTablet for tablet b5cee6e894f342f4b84a2a06f6d6659a (DEFAULT_TABLE table=test-table [id=a9ddb7932fa74efa9811e15200181806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.254347 10349 catalog_manager.cc:4841] TS e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741): tablet 1444b6a3df694ac8be4d22e2ecc83b85 (table test-table [id=b8dd3c822c494a1c897588462a638c85]) successfully deleted
I20260924 10:35:24.254459 10876 tablet_bootstrap.cc:492] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: No bootstrap required, opened a new log
I20260924 10:35:24.254534 10876 ts_tablet_manager.cc:1397] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:35:24.254658 10877 tablet_bootstrap.cc:654] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.254976 10876 raft_consensus.cc:348] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.255079 10876 raft_consensus.cc:374] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.255108 10876 raft_consensus.cc:729] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c472bc605b4a7ea9b0e6cd974de1c1, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.255239 10876 consensus_queue.cc:260] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.255350 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cee6e894f342f4b84a2a06f6d6659a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.255514 10876 ts_tablet_manager.cc:1428] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:35:24.256673 10877 tablet_bootstrap.cc:492] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: No bootstrap required, opened a new log
I20260924 10:35:24.256758 10877 ts_tablet_manager.cc:1397] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:35:24.257493 10877 raft_consensus.cc:348] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.257568 10878 tablet_bootstrap.cc:492] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Bootstrap starting.
I20260924 10:35:24.257599 10877 raft_consensus.cc:374] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.257632 10877 raft_consensus.cc:729] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bc5048aa7f4876b9614b1f2b967fa3, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.257742 10877 consensus_queue.cc:260] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.257977 10877 ts_tablet_manager.cc:1428] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:24.258553 10878 tablet_bootstrap.cc:654] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.259634 10898 ts_tablet_manager.cc:1918] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:24.259734 10898 log.cc:1196] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/1444b6a3df694ac8be4d22e2ecc83b85
I20260924 10:35:24.260092 10898 ts_tablet_manager.cc:1939] T 1444b6a3df694ac8be4d22e2ecc83b85 P 121d57765a85492ab71c703f6870058e: Deleting consensus metadata
I20260924 10:35:24.260393 10878 tablet_bootstrap.cc:492] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: No bootstrap required, opened a new log
I20260924 10:35:24.260483 10878 ts_tablet_manager.cc:1397] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 10:35:24.260881 10878 raft_consensus.cc:348] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.260994 10878 raft_consensus.cc:374] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.261031 10878 raft_consensus.cc:729] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d57765a85492ab71c703f6870058e, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.261361 10351 catalog_manager.cc:4841] TS 121d57765a85492ab71c703f6870058e (127.2.193.131:41669): tablet 1444b6a3df694ac8be4d22e2ecc83b85 (table test-table [id=b8dd3c822c494a1c897588462a638c85]) successfully deleted
I20260924 10:35:24.261148 10878 consensus_queue.cc:260] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.262027 10878 ts_tablet_manager.cc:1428] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260924 10:35:24.263862 10569 tablet.cc:2369] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:35:24.481654 10849 tablet.cc:2369] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:35:24.499897 10709 tablet.cc:2369] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:35:24.607347 10882 raft_consensus.cc:482] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:24.607589 10882 raft_consensus.cc:504] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.608155 10882 leader_election.cc:288] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919), e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741)
I20260924 10:35:24.608683 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cee6e894f342f4b84a2a06f6d6659a" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" is_pre_election: true
I20260924 10:35:24.608731 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cee6e894f342f4b84a2a06f6d6659a" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" is_pre_election: true
I20260924 10:35:24.608821 10522 raft_consensus.cc:2454] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:24.608911 10662 raft_consensus.cc:2454] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:24.609205 10733 leader_election.cc:302] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, d1bc5048aa7f4876b9614b1f2b967fa3; no voters:
I20260924 10:35:24.609390 10882 raft_consensus.cc:2781] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:24.609444 10882 raft_consensus.cc:482] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:24.609506 10882 raft_consensus.cc:3037] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.610503 10882 raft_consensus.cc:504] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.610812 10882 leader_election.cc:288] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Requested vote from peers d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919), e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741)
I20260924 10:35:24.611212 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cee6e894f342f4b84a2a06f6d6659a" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
I20260924 10:35:24.611212 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cee6e894f342f4b84a2a06f6d6659a" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3"
I20260924 10:35:24.611346 10662 raft_consensus.cc:3037] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.611357 10522 raft_consensus.cc:3037] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.612218 10522 raft_consensus.cc:2454] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:24.612327 10662 raft_consensus.cc:2454] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:24.612495 10733 leader_election.cc:302] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, d1bc5048aa7f4876b9614b1f2b967fa3; no voters:
I20260924 10:35:24.612663 10882 raft_consensus.cc:2781] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:24.612764 10882 raft_consensus.cc:686] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 1 LEADER]: Becoming Leader. State: Replica: 121d57765a85492ab71c703f6870058e, State: Running, Role: LEADER
I20260924 10:35:24.612907 10882 consensus_queue.cc:237] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.614554 10364 catalog_manager.cc:5495] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e reported cstate change: term changed from 0 to 1, leader changed from <none> to 121d57765a85492ab71c703f6870058e (127.2.193.131). New cstate: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } health_report { overall_health: HEALTHY } } }
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246123, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.190@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:24.647297 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.638906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40939 (local address 127.0.0.1:60762)
0924 10:35:24.639209 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.639232 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.639336 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.639654 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.639657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.640079 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:24.640179 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.640185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.640955 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.640957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.641612 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.641621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.642061 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.642064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.642128 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.642678 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:24.642690 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:24.644154 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:24.645799 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:24.645808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:24.645814 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:24.646065 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:24.646298 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:24.646301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:24.646302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:24.646379 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:24.646706 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:24.646711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:24.646814 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.646966 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.647102 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260924 10:35:24.647310 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.639005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.2.193.190:40939)
0924 10:35:24.639405 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.639411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.639425 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:24.639461 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.639464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.639503 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.639587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:24.640305 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.640836 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.641723 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.641937 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.642024 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.644300 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:24.644303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:24.644306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:24.644326 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:24.645703 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:24.646150 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:24.646152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:24.646154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:24.646224 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:24.646451 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:24.646453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:24.646454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:24.646621 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:24.646692 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.647020 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.647162 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":339,"thread_start_us":115,"threads_started":1}
I20260924 10:35:24.651448 10364 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:60762:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:35:24.652221 10364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:24.657269 10642 tablet_service.cc:1467] Processing CreateTablet for tablet 39e102de928346c0b43b39f6c9bf47a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=760fcd763bb44f518e464703c050447a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:35:24.657266 10502 tablet_service.cc:1467] Processing CreateTablet for tablet 39e102de928346c0b43b39f6c9bf47a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=760fcd763bb44f518e464703c050447a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:35:24.657711 10502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e102de928346c0b43b39f6c9bf47a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.657720 10642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e102de928346c0b43b39f6c9bf47a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.660452 10877 tablet_bootstrap.cc:492] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: Bootstrap starting.
I20260924 10:35:24.660470 10782 tablet_service.cc:1467] Processing CreateTablet for tablet 39e102de928346c0b43b39f6c9bf47a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=760fcd763bb44f518e464703c050447a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:35:24.660861 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e102de928346c0b43b39f6c9bf47a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.661397 10877 tablet_bootstrap.cc:654] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.662647 10877 tablet_bootstrap.cc:492] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: No bootstrap required, opened a new log
I20260924 10:35:24.662755 10877 ts_tablet_manager.cc:1397] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:24.663219 10877 raft_consensus.cc:348] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.663338 10877 raft_consensus.cc:374] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.663378 10877 raft_consensus.cc:729] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bc5048aa7f4876b9614b1f2b967fa3, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.663479 10878 tablet_bootstrap.cc:492] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: Bootstrap starting.
I20260924 10:35:24.663516 10877 consensus_queue.cc:260] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.663599 10876 tablet_bootstrap.cc:492] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: Bootstrap starting.
I20260924 10:35:24.663838 10877 ts_tablet_manager.cc:1428] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:24.664063 10884 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.664201 10884 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:35:24.664391 10878 tablet_bootstrap.cc:654] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.664445 10876 tablet_bootstrap.cc:654] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.665616 10878 tablet_bootstrap.cc:492] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: No bootstrap required, opened a new log
I20260924 10:35:24.665704 10878 ts_tablet_manager.cc:1397] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:24.666009 10876 tablet_bootstrap.cc:492] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: No bootstrap required, opened a new log
I20260924 10:35:24.666076 10876 ts_tablet_manager.cc:1397] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 10:35:24.666167 10878 raft_consensus.cc:348] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.666304 10878 raft_consensus.cc:374] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.666342 10878 raft_consensus.cc:729] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d57765a85492ab71c703f6870058e, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.666479 10878 consensus_queue.cc:260] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.666532 10876 raft_consensus.cc:348] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.666672 10876 raft_consensus.cc:374] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.666706 10876 raft_consensus.cc:729] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c472bc605b4a7ea9b0e6cd974de1c1, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.666759 10878 ts_tablet_manager.cc:1428] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:35:24.666702 10882 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.666821 10876 consensus_queue.cc:260] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.666903 10882 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:35:24.667089 10876 ts_tablet_manager.cc:1428] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:24.667363 10883 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.667472 10883 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:35:24.708871 10882 raft_consensus.cc:482] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:24.709014 10882 raft_consensus.cc:504] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.709506 10882 leader_election.cc:288] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919), e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741)
I20260924 10:35:24.710072 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e102de928346c0b43b39f6c9bf47a2" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" is_pre_election: true
I20260924 10:35:24.710085 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e102de928346c0b43b39f6c9bf47a2" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" is_pre_election: true
I20260924 10:35:24.710234 10662 raft_consensus.cc:2454] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:24.710255 10522 raft_consensus.cc:2454] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:24.710564 10733 leader_election.cc:302] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, e6c472bc605b4a7ea9b0e6cd974de1c1; no voters:
I20260924 10:35:24.710775 10882 raft_consensus.cc:2781] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:24.710835 10882 raft_consensus.cc:482] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:24.710863 10882 raft_consensus.cc:3037] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.711901 10882 raft_consensus.cc:504] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.712250 10882 leader_election.cc:288] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Requested vote from peers d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919), e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741)
I20260924 10:35:24.712651 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e102de928346c0b43b39f6c9bf47a2" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3"
I20260924 10:35:24.712778 10522 raft_consensus.cc:3037] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.712802 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e102de928346c0b43b39f6c9bf47a2" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
I20260924 10:35:24.712965 10662 raft_consensus.cc:3037] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.713837 10522 raft_consensus.cc:2454] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:24.714016 10662 raft_consensus.cc:2454] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:24.714183 10733 leader_election.cc:302] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, d1bc5048aa7f4876b9614b1f2b967fa3; no voters:
I20260924 10:35:24.714383 10882 raft_consensus.cc:2781] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:24.714486 10882 raft_consensus.cc:686] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [term 1 LEADER]: Becoming Leader. State: Replica: 121d57765a85492ab71c703f6870058e, State: Running, Role: LEADER
I20260924 10:35:24.714653 10882 consensus_queue.cc:237] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.715328 10886 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: TxnStatusTablet state changed. Reason: New leader 121d57765a85492ab71c703f6870058e. Latest consensus state: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.715442 10886 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:35:24.715667 10916 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:35:24.715909 10916 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:35:24.716477 10365 catalog_manager.cc:5495] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e reported cstate change: term changed from 0 to 1, leader changed from <none> to 121d57765a85492ab71c703f6870058e (127.2.193.131). New cstate: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } health_report { overall_health: HEALTHY } } }
I20260924 10:35:24.728606 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.724989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.2.193.131:41669)
0924 10:35:24.725437 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.725441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.725452 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:35:24.725489 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.725490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.725530 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.725630 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:24.725899 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.726464 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.727415 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.727648 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.727761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.728068 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:24.728198 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:24.728265 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.728374 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.728427 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":78,"threads_started":1}
I20260924 10:35:24.728603 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.724913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41669 (local address 127.0.0.1:40170)
0924 10:35:24.725069 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.725088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.725228 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.725651 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.725654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.725662 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:35:24.725787 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.725793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.726584 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.726588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.727287 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.727298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.727764 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.727768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.727877 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.727929 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:35:24.728280 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:35:24.728297 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.728363 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.728457 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 10:35:24.731621 10662 raft_consensus.cc:1261] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.731714 10522 raft_consensus.cc:1261] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.732158 10886 consensus_queue.cc:1035] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.732371 10882 consensus_queue.cc:1035] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.733708 10884 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: TxnStatusTablet state changed. Reason: New leader 121d57765a85492ab71c703f6870058e. Latest consensus state: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.733738 10883 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: TxnStatusTablet state changed. Reason: New leader 121d57765a85492ab71c703f6870058e. Latest consensus state: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.733835 10884 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:35:24.733845 10883 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:35:24.735985 10882 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.736114 10882 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:35:24.736078 10884 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.736188 10884 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P d1bc5048aa7f4876b9614b1f2b967fa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:35:24.736229 10882 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.736306 10882 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P 121d57765a85492ab71c703f6870058e: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:35:24.737272 10883 tablet_replica.cc:439] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } } }
I20260924 10:35:24.737401 10883 tablet_replica.cc:442] T 39e102de928346c0b43b39f6c9bf47a2 P e6c472bc605b4a7ea9b0e6cd974de1c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:35:24.752844 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.748921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:37261 (local address 127.2.193.131:41669)
0924 10:35:24.749303 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.749308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.749327 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:35:24.749380 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.749383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.749432 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.749510 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:24.749743 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.750518 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.751687 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.752001 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.752065 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.752411 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:24.752490 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:24.752528 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.752653 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.752705 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260924 10:35:24.753304 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.748848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41669 (local address 127.2.193.131:37261)
0924 10:35:24.749141 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.749157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.749289 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.749528 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.749530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.749537 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:35:24.749643 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.749649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.750634 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.750637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.751568 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.751577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.752158 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.752161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.752232 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.752275 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:35:24.752572 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:35:24.752576 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.753082 (+ 506us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.753161 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":109,"threads_started":1}
I20260924 10:35:24.767992 10662 raft_consensus.cc:1261] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.768033 10522 raft_consensus.cc:1261] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.768432 10925 consensus_queue.cc:1035] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.768582 10886 consensus_queue.cc:1035] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.807894 10365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:24.808084 10365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:24.810210 10365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Sending DeleteTablet for 3 replicas of tablet b5cee6e894f342f4b84a2a06f6d6659a
I20260924 10:35:24.810793 10502 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cee6e894f342f4b84a2a06f6d6659a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40726
I20260924 10:35:24.810952 10642 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cee6e894f342f4b84a2a06f6d6659a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52992
I20260924 10:35:24.811309 10935 tablet_replica.cc:330] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: stopping tablet replica
I20260924 10:35:24.811378 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cee6e894f342f4b84a2a06f6d6659a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40160
I20260924 10:35:24.811437 10935 raft_consensus.cc:2229] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:24.811545 10935 raft_consensus.cc:2258] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.811666 10937 tablet_replica.cc:330] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: stopping tablet replica
I20260924 10:35:24.811801 10937 raft_consensus.cc:2229] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:24.812021 10937 raft_consensus.cc:2258] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.812206 10935 ts_tablet_manager.cc:1905] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.812726 10937 ts_tablet_manager.cc:1905] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.815156 10935 ts_tablet_manager.cc:1918] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:35:24.815238 10935 log.cc:1196] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/b5cee6e894f342f4b84a2a06f6d6659a
I20260924 10:35:24.815506 10935 ts_tablet_manager.cc:1939] T b5cee6e894f342f4b84a2a06f6d6659a P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting consensus metadata
I20260924 10:35:24.815969 10937 ts_tablet_manager.cc:1918] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:35:24.816052 10937 log.cc:1196] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/b5cee6e894f342f4b84a2a06f6d6659a
I20260924 10:35:24.816185 10349 catalog_manager.cc:4841] TS d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919): tablet b5cee6e894f342f4b84a2a06f6d6659a (table test-table [id=a9ddb7932fa74efa9811e15200181806]) successfully deleted
I20260924 10:35:24.816323 10937 ts_tablet_manager.cc:1939] T b5cee6e894f342f4b84a2a06f6d6659a P 121d57765a85492ab71c703f6870058e: Deleting consensus metadata
I20260924 10:35:24.817063 10351 catalog_manager.cc:4841] TS 121d57765a85492ab71c703f6870058e (127.2.193.131:41669): tablet b5cee6e894f342f4b84a2a06f6d6659a (table test-table [id=a9ddb7932fa74efa9811e15200181806]) successfully deleted
I20260924 10:35:24.819270 10936 tablet_replica.cc:330] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: stopping tablet replica
I20260924 10:35:24.819434 10936 raft_consensus.cc:2229] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:24.819586 10936 raft_consensus.cc:2258] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.820284 10936 ts_tablet_manager.cc:1905] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.823290 10936 ts_tablet_manager.cc:1918] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:35:24.823400 10936 log.cc:1196] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/b5cee6e894f342f4b84a2a06f6d6659a
I20260924 10:35:24.823635 10936 ts_tablet_manager.cc:1939] T b5cee6e894f342f4b84a2a06f6d6659a P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting consensus metadata
I20260924 10:35:24.824395 10349 catalog_manager.cc:4841] TS e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741): tablet b5cee6e894f342f4b84a2a06f6d6659a (table test-table [id=a9ddb7932fa74efa9811e15200181806]) successfully deleted
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:35:24.842917 10911 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:24.835686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.2.193.190:40939)
0924 10:35:24.835847 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.835852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.836036 (+ 184us) server_negotiation.cc:407] Connection header received
0924 10:35:24.836178 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.836181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.836228 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.836304 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:24.837149 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.837773 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.838977 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.839220 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.839309 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.842784 (+ 3475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60766: BlockingRecv error: recv got EOF from 127.0.0.1:60766 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 10:35:24 dist-test-slave-8wc7 krb5kdc[10320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM
I20260924 10:35:24.853780 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.844647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.2.193.190:40939)
0924 10:35:24.844782 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.844786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.845007 (+ 221us) server_negotiation.cc:407] Connection header received
0924 10:35:24.845200 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.845206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.845255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.845347 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:24.845965 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.846485 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.847366 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.847710 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.847827 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.850481 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:24.850485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:24.850488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:24.850522 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:24.852021 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:24.852459 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:24.852462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:24.852464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:24.852555 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:24.852830 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:24.852834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:24.852836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:24.853033 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:24.853178 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.853473 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.853603 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:35:24.856621 10365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:24.857085 10365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:24.862126 10642 tablet_service.cc:1467] Processing CreateTablet for tablet 08fe0db4b2574d14b9ef03aed4843f40 (DEFAULT_TABLE table=test-table [id=b46b9877d00d49e4982fd4db594015cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.862164 10781 tablet_service.cc:1467] Processing CreateTablet for tablet 08fe0db4b2574d14b9ef03aed4843f40 (DEFAULT_TABLE table=test-table [id=b46b9877d00d49e4982fd4db594015cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.862162 10502 tablet_service.cc:1467] Processing CreateTablet for tablet 08fe0db4b2574d14b9ef03aed4843f40 (DEFAULT_TABLE table=test-table [id=b46b9877d00d49e4982fd4db594015cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.862442 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fe0db4b2574d14b9ef03aed4843f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.862438 10642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fe0db4b2574d14b9ef03aed4843f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.862445 10502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fe0db4b2574d14b9ef03aed4843f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.865046 10876 tablet_bootstrap.cc:492] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: Bootstrap starting.
I20260924 10:35:24.865044 10878 tablet_bootstrap.cc:492] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: Bootstrap starting.
I20260924 10:35:24.865413 10877 tablet_bootstrap.cc:492] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: Bootstrap starting.
I20260924 10:35:24.866037 10878 tablet_bootstrap.cc:654] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.866345 10877 tablet_bootstrap.cc:654] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.867412 10878 tablet_bootstrap.cc:492] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: No bootstrap required, opened a new log
I20260924 10:35:24.867492 10878 ts_tablet_manager.cc:1397] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:24.867770 10876 tablet_bootstrap.cc:654] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.868016 10878 raft_consensus.cc:348] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.868139 10878 raft_consensus.cc:374] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.868172 10878 raft_consensus.cc:729] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d57765a85492ab71c703f6870058e, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.868312 10878 consensus_queue.cc:260] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.868621 10878 ts_tablet_manager.cc:1428] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:35:24.869486 10877 tablet_bootstrap.cc:492] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: No bootstrap required, opened a new log
I20260924 10:35:24.869570 10877 ts_tablet_manager.cc:1397] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:35:24.869951 10877 raft_consensus.cc:348] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.870064 10877 raft_consensus.cc:374] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.870091 10877 raft_consensus.cc:729] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bc5048aa7f4876b9614b1f2b967fa3, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.870194 10877 consensus_queue.cc:260] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.870429 10877 ts_tablet_manager.cc:1428] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:35:24.871076 10876 tablet_bootstrap.cc:492] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: No bootstrap required, opened a new log
I20260924 10:35:24.871168 10876 ts_tablet_manager.cc:1397] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 10:35:24.871649 10876 raft_consensus.cc:348] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.871795 10876 raft_consensus.cc:374] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.871834 10876 raft_consensus.cc:729] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c472bc605b4a7ea9b0e6cd974de1c1, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.871985 10876 consensus_queue.cc:260] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.872251 10876 ts_tablet_manager.cc:1428] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:35:24.882704 10884 raft_consensus.cc:482] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:24.882836 10884 raft_consensus.cc:504] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.884086 10884 leader_election.cc:288] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741), 121d57765a85492ab71c703f6870058e (127.2.193.131:41669)
I20260924 10:35:24.888959 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.884319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41669 (local address 127.2.193.129:52829)
0924 10:35:24.884732 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.884747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.884884 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.885207 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.885209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.885225 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:24.885357 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.885363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.886398 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.886401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.887648 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.887670 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.888569 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.888572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.888656 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.888676 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.888743 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.888810 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":123,"threads_started":1}
I20260924 10:35:24.889027 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.884420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:52829 (local address 127.2.193.131:41669)
0924 10:35:24.884572 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.884577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.884815 (+ 238us) server_negotiation.cc:407] Connection header received
0924 10:35:24.885017 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.885020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.885076 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.885205 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:24.885479 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.886288 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.887793 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.888325 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.888412 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.888460 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.888792 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.888898 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:35:24.889533 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.884319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:45741 (local address 127.2.193.129:46883)
0924 10:35:24.885076 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:24.885090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:24.885178 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:24.885627 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:24.885630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:24.885645 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:24.885753 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.885759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.886797 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.886800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.888209 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:24.888218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.889068 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.889073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:24.889170 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.889226 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 10:35:24.889300 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:35:24.889369 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":380,"thread_start_us":98,"threads_started":1}
I20260924 10:35:24.890136 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.884525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:46883 (local address 127.2.193.130:45741)
0924 10:35:24.884800 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.884807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.885287 (+ 480us) server_negotiation.cc:407] Connection header received
0924 10:35:24.885342 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.885345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.885398 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.885496 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:24.885917 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.886691 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.888330 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.888830 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.888957 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.889005 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.889721 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.889799 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":122,"threads_started":1}
I20260924 10:35:24.890064 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fe0db4b2574d14b9ef03aed4843f40" candidate_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d57765a85492ab71c703f6870058e" is_pre_election: true
I20260924 10:35:24.890321 10802 raft_consensus.cc:2454] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bc5048aa7f4876b9614b1f2b967fa3 in term 0.
I20260924 10:35:24.890518 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fe0db4b2574d14b9ef03aed4843f40" candidate_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" is_pre_election: true
I20260924 10:35:24.890676 10662 raft_consensus.cc:2454] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bc5048aa7f4876b9614b1f2b967fa3 in term 0.
I20260924 10:35:24.890785 10455 leader_election.cc:302] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, d1bc5048aa7f4876b9614b1f2b967fa3; no voters:
I20260924 10:35:24.891074 10884 raft_consensus.cc:2781] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:24.891136 10884 raft_consensus.cc:482] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:24.891165 10884 raft_consensus.cc:3037] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.892210 10884 raft_consensus.cc:504] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.892597 10884 leader_election.cc:288] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [CANDIDATE]: Term 1 election: Requested vote from peers e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741), 121d57765a85492ab71c703f6870058e (127.2.193.131:41669)
I20260924 10:35:24.893008 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fe0db4b2574d14b9ef03aed4843f40" candidate_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121d57765a85492ab71c703f6870058e"
I20260924 10:35:24.893041 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fe0db4b2574d14b9ef03aed4843f40" candidate_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
I20260924 10:35:24.893157 10662 raft_consensus.cc:3037] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.893159 10802 raft_consensus.cc:3037] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:24.894129 10802 raft_consensus.cc:2454] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bc5048aa7f4876b9614b1f2b967fa3 in term 1.
I20260924 10:35:24.894129 10662 raft_consensus.cc:2454] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bc5048aa7f4876b9614b1f2b967fa3 in term 1.
I20260924 10:35:24.894461 10453 leader_election.cc:302] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1bc5048aa7f4876b9614b1f2b967fa3, e6c472bc605b4a7ea9b0e6cd974de1c1; no voters:
I20260924 10:35:24.894616 10884 raft_consensus.cc:2781] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:24.894851 10884 raft_consensus.cc:686] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 LEADER]: Becoming Leader. State: Replica: d1bc5048aa7f4876b9614b1f2b967fa3, State: Running, Role: LEADER
I20260924 10:35:24.895097 10884 consensus_queue.cc:237] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.897459 10365 catalog_manager.cc:5495] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } health_report { overall_health: UNKNOWN } } }
I20260924 10:35:24.908877 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:24.905026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.2.193.129:43919)
0924 10:35:24.905349 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:24.905356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:24.905370 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:24.905441 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:24.905443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:24.905495 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:24.905597 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:24.905799 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.906348 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.907211 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:24.907549 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:24.907633 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:24.907865 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:24.908449 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:24.908540 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:24.908621 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:35:24.908682 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":119,"threads_started":1}
I20260924 10:35:24.911794 10662 raft_consensus.cc:1261] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Refusing update from remote peer d1bc5048aa7f4876b9614b1f2b967fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.911906 10802 raft_consensus.cc:1261] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Refusing update from remote peer d1bc5048aa7f4876b9614b1f2b967fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:24.912298 10884 consensus_queue.cc:1035] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.912531 10956 consensus_queue.cc:1035] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:24.923826 10365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:24.923990 10365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:24.926056 10365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Sending DeleteTablet for 3 replicas of tablet 08fe0db4b2574d14b9ef03aed4843f40
I20260924 10:35:24.926708 10642 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fe0db4b2574d14b9ef03aed4843f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52992
I20260924 10:35:24.926708 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fe0db4b2574d14b9ef03aed4843f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40160
I20260924 10:35:24.926754 10502 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fe0db4b2574d14b9ef03aed4843f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:24 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40726
I20260924 10:35:24.926898 10937 tablet_replica.cc:330] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: stopping tablet replica
I20260924 10:35:24.926898 10936 tablet_replica.cc:330] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: stopping tablet replica
I20260924 10:35:24.926930 10935 tablet_replica.cc:330] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: stopping tablet replica
I20260924 10:35:24.927011 10937 raft_consensus.cc:2229] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:24.927009 10936 raft_consensus.cc:2229] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:24.927031 10935 raft_consensus.cc:2229] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:24.927136 10937 raft_consensus.cc:2258] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.927134 10936 raft_consensus.cc:2258] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.927349 10935 raft_consensus.cc:2258] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:24.928102 10937 ts_tablet_manager.cc:1905] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.928102 10935 ts_tablet_manager.cc:1905] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.928296 10365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:24.928737 10365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:24.930630 10937 ts_tablet_manager.cc:1918] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:24.930630 10935 ts_tablet_manager.cc:1918] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:24.930702 10937 log.cc:1196] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/08fe0db4b2574d14b9ef03aed4843f40
I20260924 10:35:24.930702 10935 log.cc:1196] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/08fe0db4b2574d14b9ef03aed4843f40
I20260924 10:35:24.930967 10937 ts_tablet_manager.cc:1939] T 08fe0db4b2574d14b9ef03aed4843f40 P 121d57765a85492ab71c703f6870058e: Deleting consensus metadata
I20260924 10:35:24.930967 10935 ts_tablet_manager.cc:1939] T 08fe0db4b2574d14b9ef03aed4843f40 P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting consensus metadata
I20260924 10:35:24.931690 10351 catalog_manager.cc:4841] TS 121d57765a85492ab71c703f6870058e (127.2.193.131:41669): tablet 08fe0db4b2574d14b9ef03aed4843f40 (table test-table [id=b46b9877d00d49e4982fd4db594015cf]) successfully deleted
I20260924 10:35:24.931991 10349 catalog_manager.cc:4841] TS d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919): tablet 08fe0db4b2574d14b9ef03aed4843f40 (table test-table [id=b46b9877d00d49e4982fd4db594015cf]) successfully deleted
I20260924 10:35:24.932502 10936 ts_tablet_manager.cc:1905] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:24.934932 10936 ts_tablet_manager.cc:1918] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:24.935000 10936 log.cc:1196] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/08fe0db4b2574d14b9ef03aed4843f40
I20260924 10:35:24.935253 10936 ts_tablet_manager.cc:1939] T 08fe0db4b2574d14b9ef03aed4843f40 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting consensus metadata
I20260924 10:35:24.935911 10349 catalog_manager.cc:4841] TS e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741): tablet 08fe0db4b2574d14b9ef03aed4843f40 (table test-table [id=b46b9877d00d49e4982fd4db594015cf]) successfully deleted
I20260924 10:35:24.937681 10502 tablet_service.cc:1467] Processing CreateTablet for tablet ca5bee4220d843d5b52a67fa4daccd80 (DEFAULT_TABLE table=test-table [id=ec5835752cf44b96b3388c604cc3d839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.937788 10642 tablet_service.cc:1467] Processing CreateTablet for tablet ca5bee4220d843d5b52a67fa4daccd80 (DEFAULT_TABLE table=test-table [id=ec5835752cf44b96b3388c604cc3d839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.937943 10502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5bee4220d843d5b52a67fa4daccd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.937896 10781 tablet_service.cc:1467] Processing CreateTablet for tablet ca5bee4220d843d5b52a67fa4daccd80 (DEFAULT_TABLE table=test-table [id=ec5835752cf44b96b3388c604cc3d839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:24.938040 10642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5bee4220d843d5b52a67fa4daccd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.938131 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca5bee4220d843d5b52a67fa4daccd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:24.940182 10877 tablet_bootstrap.cc:492] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3: Bootstrap starting.
I20260924 10:35:24.940268 10876 tablet_bootstrap.cc:492] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: Bootstrap starting.
I20260924 10:35:24.941154 10877 tablet_bootstrap.cc:654] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.941243 10876 tablet_bootstrap.cc:654] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.941803 10878 tablet_bootstrap.cc:492] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: Bootstrap starting.
I20260924 10:35:24.942469 10877 tablet_bootstrap.cc:492] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3: No bootstrap required, opened a new log
I20260924 10:35:24.942559 10877 ts_tablet_manager.cc:1397] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:35:24.942682 10878 tablet_bootstrap.cc:654] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:24.943077 10877 raft_consensus.cc:348] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.943213 10877 raft_consensus.cc:374] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.943265 10877 raft_consensus.cc:729] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bc5048aa7f4876b9614b1f2b967fa3, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.943403 10877 consensus_queue.cc:260] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.943710 10877 ts_tablet_manager.cc:1428] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:35:24.943902 10876 tablet_bootstrap.cc:492] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: No bootstrap required, opened a new log
I20260924 10:35:24.944015 10876 ts_tablet_manager.cc:1397] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 10:35:24.944398 10878 tablet_bootstrap.cc:492] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: No bootstrap required, opened a new log
I20260924 10:35:24.944483 10878 ts_tablet_manager.cc:1397] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 10:35:24.944494 10876 raft_consensus.cc:348] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.944602 10876 raft_consensus.cc:374] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.944640 10876 raft_consensus.cc:729] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c472bc605b4a7ea9b0e6cd974de1c1, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.944759 10876 consensus_queue.cc:260] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.944883 10878 raft_consensus.cc:348] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.944984 10878 raft_consensus.cc:374] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:24.945020 10878 raft_consensus.cc:729] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121d57765a85492ab71c703f6870058e, State: Initialized, Role: FOLLOWER
I20260924 10:35:24.945062 10876 ts_tablet_manager.cc:1428] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:35:24.945155 10878 consensus_queue.cc:260] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:24.945458 10878 ts_tablet_manager.cc:1428] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:35:25.075523 10886 raft_consensus.cc:482] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:25.075670 10886 raft_consensus.cc:504] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:25.076201 10886 leader_election.cc:288] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919), e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741)
I20260924 10:35:25.076689 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5bee4220d843d5b52a67fa4daccd80" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" is_pre_election: true
I20260924 10:35:25.076777 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5bee4220d843d5b52a67fa4daccd80" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" is_pre_election: true
I20260924 10:35:25.076851 10522 raft_consensus.cc:2454] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:25.076920 10662 raft_consensus.cc:2454] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 0.
I20260924 10:35:25.077195 10733 leader_election.cc:302] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, d1bc5048aa7f4876b9614b1f2b967fa3; no voters:
I20260924 10:35:25.077410 10886 raft_consensus.cc:2781] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:25.077471 10886 raft_consensus.cc:482] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:25.077529 10886 raft_consensus.cc:3037] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:25.078552 10886 raft_consensus.cc:504] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:25.078945 10886 leader_election.cc:288] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Requested vote from peers d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919), e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741)
I20260924 10:35:25.079340 10522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5bee4220d843d5b52a67fa4daccd80" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3"
I20260924 10:35:25.079392 10662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca5bee4220d843d5b52a67fa4daccd80" candidate_uuid: "121d57765a85492ab71c703f6870058e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1"
I20260924 10:35:25.079478 10522 raft_consensus.cc:3037] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:25.079545 10662 raft_consensus.cc:3037] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:25.080384 10522 raft_consensus.cc:2454] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:25.080659 10662 raft_consensus.cc:2454] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 121d57765a85492ab71c703f6870058e in term 1.
I20260924 10:35:25.080725 10733 leader_election.cc:302] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121d57765a85492ab71c703f6870058e, d1bc5048aa7f4876b9614b1f2b967fa3; no voters:
I20260924 10:35:25.081027 10886 raft_consensus.cc:2781] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:25.081142 10886 raft_consensus.cc:686] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 1 LEADER]: Becoming Leader. State: Replica: 121d57765a85492ab71c703f6870058e, State: Running, Role: LEADER
I20260924 10:35:25.081314 10886 consensus_queue.cc:237] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } }
I20260924 10:35:25.083225 10365 catalog_manager.cc:5495] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e reported cstate change: term changed from 0 to 1, leader changed from <none> to 121d57765a85492ab71c703f6870058e (127.2.193.131). New cstate: current_term: 1 leader_uuid: "121d57765a85492ab71c703f6870058e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "121d57765a85492ab71c703f6870058e" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41669 } health_report { overall_health: HEALTHY } } }
I20260924 10:35:25.103344 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:25.100538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.2.193.131:41669)
0924 10:35:25.100655 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:25.100658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:25.100694 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:35:25.100816 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:25.100818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:25.100860 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:25.100920 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:25.101119 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:25.101673 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:25.102461 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:25.102677 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:25.102752 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:25.102994 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:25.103061 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:25.103114 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:25.103191 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 10:35:25.103237 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:35:25.109373 10661 raft_consensus.cc:1261] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:25.109349 10522 raft_consensus.cc:1261] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Refusing update from remote peer 121d57765a85492ab71c703f6870058e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:25.109720 10882 consensus_queue.cc:1035] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bc5048aa7f4876b9614b1f2b967fa3" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 43919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:25.109864 10968 consensus_queue.cc:1035] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c472bc605b4a7ea9b0e6cd974de1c1" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:25.138113 10363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:25.138267 10363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:25.140118 10363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 172536804a9b4669bd5d016fee4025dc: Sending DeleteTablet for 3 replicas of tablet ca5bee4220d843d5b52a67fa4daccd80
I20260924 10:35:25.140715 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5bee4220d843d5b52a67fa4daccd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:25 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40160
I20260924 10:35:25.140735 10502 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5bee4220d843d5b52a67fa4daccd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:25 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:40726
I20260924 10:35:25.140738 10642 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5bee4220d843d5b52a67fa4daccd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:25 UTC) from {username='oryx', principal='oryx/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52992
I20260924 10:35:25.140913 10935 tablet_replica.cc:330] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3: stopping tablet replica
I20260924 10:35:25.140991 10937 tablet_replica.cc:330] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: stopping tablet replica
I20260924 10:35:25.141091 10935 raft_consensus.cc:2229] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:25.141144 10937 raft_consensus.cc:2229] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:25.141253 10935 raft_consensus.cc:2258] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:25.141382 10937 raft_consensus.cc:2258] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:25.141578 10936 tablet_replica.cc:330] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: stopping tablet replica
I20260924 10:35:25.141693 10936 raft_consensus.cc:2229] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:25.141786 10936 raft_consensus.cc:2258] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:25.142030 10935 ts_tablet_manager.cc:1905] T ca5bee4220d843d5b52a67fa4daccd80 P d1bc5048aa7f4876b9614b1f2b967fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:25.142369 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 10437
I20260924 10:35:25.142409 10937 ts_tablet_manager.cc:1905] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:25.144712 10937 ts_tablet_manager.cc:1918] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:35:25.144784 10937 log.cc:1196] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/ca5bee4220d843d5b52a67fa4daccd80
I20260924 10:35:25.145027 10937 ts_tablet_manager.cc:1939] T ca5bee4220d843d5b52a67fa4daccd80 P 121d57765a85492ab71c703f6870058e: Deleting consensus metadata
I20260924 10:35:25.145747 10936 ts_tablet_manager.cc:1905] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:25.146025 10351 catalog_manager.cc:4841] TS 121d57765a85492ab71c703f6870058e (127.2.193.131:41669): tablet ca5bee4220d843d5b52a67fa4daccd80 (table test-table [id=ec5835752cf44b96b3388c604cc3d839]) successfully deleted
I20260924 10:35:25.147953 10936 ts_tablet_manager.cc:1918] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:35:25.148020 10936 log.cc:1196] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/ca5bee4220d843d5b52a67fa4daccd80
I20260924 10:35:25.148262 10936 ts_tablet_manager.cc:1939] T ca5bee4220d843d5b52a67fa4daccd80 P e6c472bc605b4a7ea9b0e6cd974de1c1: Deleting consensus metadata
I20260924 10:35:25.148924 10349 catalog_manager.cc:4841] TS e6c472bc605b4a7ea9b0e6cd974de1c1 (127.2.193.130:45741): tablet ca5bee4220d843d5b52a67fa4daccd80 (table test-table [id=ec5835752cf44b96b3388c604cc3d839]) successfully deleted
I20260924 10:35:25.150564 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 10577
W20260924 10:35:25.150776 10349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:43919 (error 108)
I20260924 10:35:25.155012 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:25.154590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:43919 (local address 127.0.0.1:40730)
0924 10:35:25.154853 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:25.154924 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:43919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":110,"threads_started":1}
W20260924 10:35:25.155153 10349 catalog_manager.cc:4584] TS d1bc5048aa7f4876b9614b1f2b967fa3 (127.2.193.129:43919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca5bee4220d843d5b52a67fa4daccd80: Network error: Client connection negotiation failed: client connection to 127.2.193.129:43919: connect: Connection refused (error 111)
I20260924 10:35:25.157080 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 10717
I20260924 10:35:25.163890 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 10333
2026-09-24T10:35:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3333 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:25 dist-test-slave-8wc7 krb5kdc[10975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:25 dist-test-slave-8wc7 krb5kdc[10975](info): set up 2 sockets
Sep 24 10:35:25 dist-test-slave-8wc7 krb5kdc[10975](info): commencing operation
krb5kdc: starting...
W20260924 10:35:26.201995 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:26 dist-test-slave-8wc7 krb5kdc[10975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:26.328378 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46849
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:26.439772 10988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:26.440125 10988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:26.440208 10988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:26.444417 10988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:26.444527 10988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:26.444561 10988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:26.444600 10988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:26.444633 10988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:26.450198 10988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:26.451366 10988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:26.452450 10988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:26.461279 10993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:26.461361 10988 server_base.cc:1034] running on GCE node
W20260924 10:35:26.461279 10994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:26.461283 10996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:26.462191 10988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:26.463335 10988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:26.464522 10988 hybrid_clock.cc:648] HybridClock initialized: now 1790246126464506 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T10:35:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1310 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:26 dist-test-slave-8wc7 krb5kdc[11001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:26 dist-test-slave-8wc7 krb5kdc[11001](info): set up 2 sockets
Sep 24 10:35:26 dist-test-slave-8wc7 krb5kdc[11001](info): commencing operation
krb5kdc: starting...
W20260924 10:35:27.505903 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:27 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:27.627629 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46035
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:27.741291 11014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:27.741637 11014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:27.741744 11014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:27.745822 11014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:27.745909 11014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:27.745945 11014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:27.746002 11014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:27.746039 11014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:27.751575 11014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:27.752748 11014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:27.753947 11014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:27.762773 11019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:27.762771 11022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:27.762774 11020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:27.762902 11014 server_base.cc:1034] running on GCE node
I20260924 10:35:27.763691 11014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:27.764865 11014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:27.766058 11014 hybrid_clock.cc:648] HybridClock initialized: now 1790246127766020 us; error 61 us; skew 500 ppm
Sep 24 10:35:27 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246127, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:27.774593 11014 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:27.775818 11014 webserver.cc:460] Webserver started at http://127.2.193.190:40117/ using document root <none> and password file <none>
I20260924 10:35:27.776436 11014 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:27.776544 11014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:27.776774 11014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:27.779035 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "54febb5524d04ce998a5c92b685166eb"
format_stamp: "Formatted at 2026-09-24 10:35:27 on dist-test-slave-8wc7"
server_key: "82ec4012a3e9fe321ecc4cee8521bf4f"
server_key_iv: "99a503e9b960e3b65c7ab89890d2b04e"
server_key_version: "encryptionkey@0"
I20260924 10:35:27.779646 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "54febb5524d04ce998a5c92b685166eb"
format_stamp: "Formatted at 2026-09-24 10:35:27 on dist-test-slave-8wc7"
server_key: "82ec4012a3e9fe321ecc4cee8521bf4f"
server_key_iv: "99a503e9b960e3b65c7ab89890d2b04e"
server_key_version: "encryptionkey@0"
I20260924 10:35:27.783679 11014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:27.786381 11029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:27.787851 11014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:27.788008 11014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "54febb5524d04ce998a5c92b685166eb"
format_stamp: "Formatted at 2026-09-24 10:35:27 on dist-test-slave-8wc7"
server_key: "82ec4012a3e9fe321ecc4cee8521bf4f"
server_key_iv: "99a503e9b960e3b65c7ab89890d2b04e"
server_key_version: "encryptionkey@0"
I20260924 10:35:27.788138 11014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:27.803956 11014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:27.807037 11014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:27.807291 11014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:27.814899 11014 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:46035
I20260924 10:35:27.814916 11081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:46035 every 8 connection(s)
I20260924 10:35:27.815960 11014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:27.818883 11082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:27.824474 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11014
I20260924 10:35:27.824623 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:27.824831 11082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Bootstrap starting.
I20260924 10:35:27.824939 2822 external_mini_cluster.cc:1428] Setting key a8c66a3889c3d41834e666c4af0b9565
I20260924 10:35:27.827502 11082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:27.828263 11082 log.cc:824] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:27.830240 11082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: No bootstrap required, opened a new log
Sep 24 10:35:27 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:27.833957 11082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:27.834276 11082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:27.834364 11082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54febb5524d04ce998a5c92b685166eb, State: Initialized, Role: FOLLOWER
I20260924 10:35:27.834860 11082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:27.835016 11082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:27.835083 11082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:27.835182 11082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:27.836078 11082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:27.836443 11082 leader_election.cc:302] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54febb5524d04ce998a5c92b685166eb; no voters:
I20260924 10:35:27.836782 11082 leader_election.cc:288] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:27.836853 11087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:27.837054 11087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 LEADER]: Becoming Leader. State: Replica: 54febb5524d04ce998a5c92b685166eb, State: Running, Role: LEADER
I20260924 10:35:27.837430 11087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:27.838142 11082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:27.839192 11089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54febb5524d04ce998a5c92b685166eb. Latest consensus state: current_term: 1 leader_uuid: "54febb5524d04ce998a5c92b685166eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } } }
I20260924 10:35:27.839325 11089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:27.839708 11096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:27.840304 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:27.826454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.2.193.190:46035)
0924 10:35:27.826927 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:27.826942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:27.826979 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:35:27.827759 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:27.827782 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:27.828143 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:27.828499 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:27.829362 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:27.830369 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:27.831269 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:27.831754 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:27.831942 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:27.834698 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:27.834704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:27.834718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:27.834749 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:27.836659 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:27.837268 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:27.837274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:27.837280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:27.837433 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:27.837757 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:27.837760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:27.837761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:27.838159 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:27.838415 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:27.838883 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 10:35:27.839256 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":141,"threads_started":1}
I20260924 10:35:27.840797 11088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54febb5524d04ce998a5c92b685166eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } } }
I20260924 10:35:27.840900 11088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:27.843123 11096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:27.849102 11096 catalog_manager.cc:1347] Generated new cluster ID: d5a842b2173b48fbb4c322987c844911
I20260924 10:35:27.849296 11096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:27.854683 11096 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:27.855692 11096 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:27.861016 11096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Generated new TSK 0
I20260924 10:35:27.861857 11096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:35:27.895952 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46035
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:28.014853 11108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:28.015159 11108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:28.015230 11108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:28.019445 11108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:28.019547 11108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:28.019639 11108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:28.025143 11108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:28.026373 11108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:28.027480 11108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:28.037349 11116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:28.037367 11113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:28.037444 11108 server_base.cc:1034] running on GCE node
W20260924 10:35:28.037349 11114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:28.038048 11108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:28.038818 11108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:28.040015 11108 hybrid_clock.cc:648] HybridClock initialized: now 1790246128040001 us; error 36 us; skew 500 ppm
Sep 24 10:35:28 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246128, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:28.042968 11108 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:35:28.043915 11108 webserver.cc:460] Webserver started at http://127.2.193.129:33213/ using document root <none> and password file <none>
I20260924 10:35:28.044529 11108 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:28.044626 11108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:28.044876 11108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:28.047130 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "186de9e053ff435ebcbde407d5df8dfb"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "aea8206cb3d1e8cb9a92848522a133d3"
server_key_iv: "4c24239e2f6142f87d40bfcd89a8de11"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.047746 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "186de9e053ff435ebcbde407d5df8dfb"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "aea8206cb3d1e8cb9a92848522a133d3"
server_key_iv: "4c24239e2f6142f87d40bfcd89a8de11"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.051633 11108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:28.054155 11123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.055488 11108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:28.055647 11108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "186de9e053ff435ebcbde407d5df8dfb"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "aea8206cb3d1e8cb9a92848522a133d3"
server_key_iv: "4c24239e2f6142f87d40bfcd89a8de11"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.055778 11108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:28.094188 11108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:28.097900 11108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:28.098125 11108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:28.098763 11108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:28.099685 11108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:28.099756 11108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.099830 11108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:28.099881 11108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.109078 11108 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38131
I20260924 10:35:28.109102 11236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38131 every 8 connection(s)
I20260924 10:35:28.110072 11108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:28.113435 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11108
I20260924 10:35:28.113571 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:28.113847 2822 external_mini_cluster.cc:1428] Setting key 84820a4699fbc2e1b0b8aeaf088b19f9
Sep 24 10:35:28 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246128, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:35:28.126454 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:28.111932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:32837 (local address 127.2.193.190:46035)
0924 10:35:28.112078 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:28.112082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:28.113247 (+ 1165us) server_negotiation.cc:407] Connection header received
0924 10:35:28.114312 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:28.114314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:28.114361 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:28.114487 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:28.115915 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.116766 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.118634 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.118866 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.118954 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:28.122352 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:28.122355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:28.122357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:28.122384 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:28.124150 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:28.124879 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:28.124882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:28.124883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:28.124963 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:28.125367 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:28.125370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:28.125371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:28.125559 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:28.125639 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:28.126178 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 10:35:28.126303 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:28.127418 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:28.112353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:32837)
0924 10:35:28.113071 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:28.113116 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:28.114086 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:28.114673 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:28.114682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:28.115200 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:28.115762 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:28.115775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.116919 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.116924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:28.118508 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:28.118523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.119034 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.119041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:28.119277 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:28.120008 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:28.120036 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:28.122155 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:28.124301 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:28.124309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:28.124326 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:28.124757 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:28.125051 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:28.125053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:28.125055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:28.125253 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:28.125672 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:28.125679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:28.125945 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 10:35:28.126111 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 10:35:28.126388 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":119,"threads_started":1}
I20260924 10:35:28.128749 11237 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46035
I20260924 10:35:28.129040 11237 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:28.129568 11237 heartbeater.cc:511] Master 127.2.193.190:46035 requested a full tablet report, sending...
I20260924 10:35:28.131283 11046 ts_manager.cc:194] Registered new tserver with Master: 186de9e053ff435ebcbde407d5df8dfb (127.2.193.129:38131)
I20260924 10:35:28.133205 11046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:32837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:35:28.144413 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46035
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:28.262868 11242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:28.263166 11242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:28.263250 11242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:28.267519 11242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:28.267599 11242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:28.267705 11242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:28.273360 11242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:28.274518 11242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:28.275671 11242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:28.285511 11248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:28.285568 11247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:28.285668 11242 server_base.cc:1034] running on GCE node
W20260924 10:35:28.285511 11250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:28.286321 11242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:28.287005 11242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:28.288167 11242 hybrid_clock.cc:648] HybridClock initialized: now 1790246128288135 us; error 45 us; skew 500 ppm
Sep 24 10:35:28 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246128, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:28.291160 11242 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:35:28.292212 11242 webserver.cc:460] Webserver started at http://127.2.193.130:41383/ using document root <none> and password file <none>
I20260924 10:35:28.292851 11242 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:28.292943 11242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:28.293219 11242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:28.295342 11242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "85cc30395de64ce995fc9aa53ef0c902"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "c0de6e0024469a248b6d8632bd9bfec1"
server_key_iv: "e4cc9e93a21d58cb0554d8c0d06055a9"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.295943 11242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "85cc30395de64ce995fc9aa53ef0c902"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "c0de6e0024469a248b6d8632bd9bfec1"
server_key_iv: "e4cc9e93a21d58cb0554d8c0d06055a9"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.299891 11242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:28.302497 11257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.303766 11242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:28.303905 11242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "85cc30395de64ce995fc9aa53ef0c902"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "c0de6e0024469a248b6d8632bd9bfec1"
server_key_iv: "e4cc9e93a21d58cb0554d8c0d06055a9"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.304031 11242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:28.330129 11242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:28.333808 11242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:28.334033 11242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:28.334673 11242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:28.335554 11242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:28.335625 11242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.335698 11242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:28.335748 11242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.345358 11242 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:45605
I20260924 10:35:28.345413 11370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:45605 every 8 connection(s)
I20260924 10:35:28.346336 11242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:28.351267 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11242
I20260924 10:35:28.351372 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:28.351650 2822 external_mini_cluster.cc:1428] Setting key eaf4442a0e6cb00ea147ac1897b1d4eb
Sep 24 10:35:28 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246128, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:28.361722 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:28.348243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:42223 (local address 127.2.193.190:46035)
0924 10:35:28.348408 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:28.348413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:28.349380 (+ 967us) server_negotiation.cc:407] Connection header received
0924 10:35:28.350617 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:28.350620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:28.350668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:28.350747 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:28.352103 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.352774 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.354183 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.354582 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.354716 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:28.357576 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:28.357580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:28.357583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:28.357614 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:28.359397 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:28.359948 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:28.359953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:28.359955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:28.360054 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:28.360454 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:28.360458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:28.360459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:28.360659 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:28.360770 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:28.361415 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 10:35:28.361554 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:28.362687 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:28.348688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:42223)
0924 10:35:28.349166 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:28.349266 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:28.350418 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:28.350913 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:28.350919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:28.351293 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:28.351921 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:28.351938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.353047 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.353052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:28.354053 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:28.354063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.354766 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.354773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:28.355008 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:28.355668 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:28.355700 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:28.357384 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:28.359530 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:28.359536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:28.359555 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:28.359826 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:28.360144 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:28.360163 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:28.360166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:28.360321 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:28.360785 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:28.360794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:28.361114 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 10:35:28.361359 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 10:35:28.361711 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260924 10:35:28.363895 11371 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
I20260924 10:35:28.364223 11371 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:28.364723 11371 heartbeater.cc:511] Master 127.2.193.190:46035 requested a full tablet report, sending...
I20260924 10:35:28.365881 11046 ts_manager.cc:194] Registered new tserver with Master: 85cc30395de64ce995fc9aa53ef0c902 (127.2.193.130:45605)
I20260924 10:35:28.366485 11046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:42223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:35:28.380199 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46035
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:28.495136 11376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:28.495441 11376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:28.495486 11376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:28.499671 11376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:28.499751 11376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:28.499814 11376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:28.505192 11376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:28.506408 11376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:28.507541 11376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:28.517578 11376 server_base.cc:1034] running on GCE node
W20260924 10:35:28.517531 11381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:28.517539 11382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:28.517531 11384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:28.518215 11376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:28.518883 11376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:28.520079 11376 hybrid_clock.cc:648] HybridClock initialized: now 1790246128520060 us; error 31 us; skew 500 ppm
Sep 24 10:35:28 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246128, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:28.522925 11376 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:35:28.523897 11376 webserver.cc:460] Webserver started at http://127.2.193.131:43185/ using document root <none> and password file <none>
I20260924 10:35:28.524495 11376 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:28.524590 11376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:28.524837 11376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:28.527105 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "e41a0239c2e04ff49336554dfc1665e2"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "22bf61b1c46f80f15351de206ab501e3"
server_key_iv: "0adbf0aedf69f66036dd84d64d874518"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.527720 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "e41a0239c2e04ff49336554dfc1665e2"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "22bf61b1c46f80f15351de206ab501e3"
server_key_iv: "0adbf0aedf69f66036dd84d64d874518"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.531740 11376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:28.534260 11391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.535495 11376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:28.535640 11376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "e41a0239c2e04ff49336554dfc1665e2"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "22bf61b1c46f80f15351de206ab501e3"
server_key_iv: "0adbf0aedf69f66036dd84d64d874518"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.535759 11376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:28.585764 11376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:28.588903 11376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:28.589128 11376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:28.589787 11376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:28.590610 11376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:28.590656 11376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.590698 11376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:28.590719 11376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.599995 11376 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:36245
I20260924 10:35:28.600018 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:36245 every 8 connection(s)
I20260924 10:35:28.601033 11376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:28.607707 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11376
I20260924 10:35:28.607875 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:28.608157 2822 external_mini_cluster.cc:1428] Setting key 08954b9bee45aadb797bf40a409f2bc9
Sep 24 10:35:28 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246128, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:28.615141 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:28.603078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:46335 (local address 127.2.193.190:46035)
0924 10:35:28.603217 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:28.603221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:28.604266 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 10:35:28.605235 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:28.605239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:28.605300 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:28.605407 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:28.606678 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.607220 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.608650 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.608901 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.608998 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:28.611961 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:28.611971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:28.611973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:28.611998 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:28.613301 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:28.613781 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:28.613784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:28.613785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:28.613857 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:28.614147 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:28.614149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:28.614150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:28.614319 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:28.614397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:28.614913 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 10:35:28.615019 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:35:28.616199 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:28.603420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:46335)
0924 10:35:28.604101 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:28.604145 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:28.604993 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:28.605610 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:28.605617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:28.606005 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:28.606520 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:28.606534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.607397 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.607404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:28.608512 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:28.608526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:28.609065 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:28.609069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:28.609332 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:28.610011 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:28.610032 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:28.611798 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:28.613407 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:28.613412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:28.613426 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:28.613695 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:28.613930 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:28.613932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:28.613934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:28.614070 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:28.614430 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:28.614436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:28.614703 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 10:35:28.614875 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 10:35:28.615146 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":114,"threads_started":1}
I20260924 10:35:28.617439 11505 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46035
I20260924 10:35:28.617713 11505 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:28.618225 11505 heartbeater.cc:511] Master 127.2.193.190:46035 requested a full tablet report, sending...
I20260924 10:35:28.619257 11046 ts_manager.cc:194] Registered new tserver with Master: e41a0239c2e04ff49336554dfc1665e2 (127.2.193.131:36245)
I20260924 10:35:28.619831 11046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:46335
I20260924 10:35:28.622352 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab.
Entry for principal oryx/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab.
WARNING: no policy specified for oryx/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab.
Entry for principal oryx/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab.
WARNING: no policy specified for oryx/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab.
Entry for principal oryx/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab.
I20260924 10:35:28.692189 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11108
I20260924 10:35:28.697743 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11242
I20260924 10:35:28.703248 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11376
I20260924 10:35:28.708359 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11014
I20260924 10:35:28.714486 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46035
--webserver_interface=127.2.193.190
--webserver_port=40117
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:28.830230 11514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:28.830597 11514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:28.830669 11514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:28.834870 11514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:28.834990 11514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:28.835028 11514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:28.835069 11514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:28.835105 11514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:28.840698 11514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:28.841939 11514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:28.843048 11514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:28.851999 11520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:28.851886 11522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:28.851886 11519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:28.852178 11514 server_base.cc:1034] running on GCE node
I20260924 10:35:28.852881 11514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:28.854110 11514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:28.855350 11514 hybrid_clock.cc:648] HybridClock initialized: now 1790246128855332 us; error 37 us; skew 500 ppm
Sep 24 10:35:28 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246128, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:28.858619 11514 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:28.859637 11514 webserver.cc:460] Webserver started at http://127.2.193.190:40117/ using document root <none> and password file <none>
I20260924 10:35:28.860244 11514 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:28.860351 11514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:28.864423 11514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:35:28.866751 11529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:28.868093 11514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:28.868244 11514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "54febb5524d04ce998a5c92b685166eb"
format_stamp: "Formatted at 2026-09-24 10:35:27 on dist-test-slave-8wc7"
server_key: "82ec4012a3e9fe321ecc4cee8521bf4f"
server_key_iv: "99a503e9b960e3b65c7ab89890d2b04e"
server_key_version: "encryptionkey@0"
I20260924 10:35:28.868826 11514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:28.894428 11514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:28.897630 11514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:28.897876 11514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:28.905280 11514 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:46035
I20260924 10:35:28.905293 11581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:46035 every 8 connection(s)
I20260924 10:35:28.906425 11514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:28.910884 11582 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:35:28.911279 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11514
I20260924 10:35:28.911746 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:38131
--local_ip_for_outbound_sockets=127.2.193.129
--tserver_master_addrs=127.2.193.190:46035
--webserver_port=33213
--webserver_interface=127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:35:28.912788 11582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Bootstrap starting.
I20260924 10:35:28.929534 11582 log.cc:824] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:28.936221 11582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:35:28.936632 11582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Bootstrap complete.
I20260924 10:35:28.940871 11582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:28.941385 11582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54febb5524d04ce998a5c92b685166eb, State: Initialized, Role: FOLLOWER
I20260924 10:35:28.941954 11582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:28.942098 11582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:28.942186 11582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:28.942298 11582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:35:28.944312 11582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:28.944664 11582 leader_election.cc:302] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54febb5524d04ce998a5c92b685166eb; no voters:
I20260924 10:35:28.944994 11582 leader_election.cc:288] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:35:28.945148 11586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:35:28.945428 11586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [term 2 LEADER]: Becoming Leader. State: Replica: 54febb5524d04ce998a5c92b685166eb, State: Running, Role: LEADER
I20260924 10:35:28.945837 11586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } }
I20260924 10:35:28.946317 11582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:28.947299 11587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "54febb5524d04ce998a5c92b685166eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } } }
I20260924 10:35:28.947428 11587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:28.947733 11588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54febb5524d04ce998a5c92b685166eb. Latest consensus state: current_term: 2 leader_uuid: "54febb5524d04ce998a5c92b685166eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54febb5524d04ce998a5c92b685166eb" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46035 } } }
I20260924 10:35:28.947811 11588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:28.948437 11595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:28.951985 11595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:28.952628 11595 catalog_manager.cc:1259] Loaded cluster ID: d5a842b2173b48fbb4c322987c844911
I20260924 10:35:28.952690 11595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:28.955047 11595 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:28.955722 11595 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54febb5524d04ce998a5c92b685166eb: Loaded TSK: 0
I20260924 10:35:28.956956 11595 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:35:29.028255 11584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:29.028574 11584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:29.028668 11584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:29.032866 11584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:29.032979 11584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:29.033078 11584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:29.038750 11584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.129.keytab
--principal=oryx/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:38131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=33213
--tserver_master_addrs=127.2.193.190:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:29.039851 11584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:29.040918 11584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:29.052624 11584 server_base.cc:1034] running on GCE node
W20260924 10:35:29.052609 11608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:29.052731 11611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:29.052619 11609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:29.053503 11584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:29.054186 11584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:29.055354 11584 hybrid_clock.cc:648] HybridClock initialized: now 1790246129055324 us; error 52 us; skew 500 ppm
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246129, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:29.058409 11584 init.cc:377] Logged in from keytab as oryx/127.2.193.129@KRBTEST.COM (short username oryx)
I20260924 10:35:29.059445 11584 webserver.cc:460] Webserver started at http://127.2.193.129:33213/ using document root <none> and password file <none>
I20260924 10:35:29.060030 11584 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:29.060145 11584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:29.064277 11584 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:35:29.066726 11618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.068264 11584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:35:29.068406 11584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "186de9e053ff435ebcbde407d5df8dfb"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "aea8206cb3d1e8cb9a92848522a133d3"
server_key_iv: "4c24239e2f6142f87d40bfcd89a8de11"
server_key_version: "encryptionkey@0"
I20260924 10:35:29.068998 11584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:29.091926 11584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:29.095472 11584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:29.095700 11584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:29.096341 11584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:29.097237 11584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:29.097311 11584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.097388 11584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:29.097440 11584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.106766 11584 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38131
I20260924 10:35:29.106791 11731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38131 every 8 connection(s)
I20260924 10:35:29.107851 11584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:29.108683 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11584
I20260924 10:35:29.109302 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:45605
--local_ip_for_outbound_sockets=127.2.193.130
--tserver_master_addrs=127.2.193.190:46035
--webserver_port=41383
--webserver_interface=127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:29.121845 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.110369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:55419)
0924 10:35:29.111407 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.111449 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.112217 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.113399 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.113407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.113852 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.114320 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.114334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.115652 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.115657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.116548 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.116558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.117419 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.117424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.117614 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.118265 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.118281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.120776 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":115,"threads_started":1}
W20260924 10:35:29.121819 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.110270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:55419 (local address 127.2.193.190:46035)
0924 10:35:29.110887 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.110907 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.111587 (+ 680us) server_negotiation.cc:407] Connection header received
0924 10:35:29.112644 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.112655 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.112968 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.113328 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.114512 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.115504 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.116699 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.117248 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.117430 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.121015 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:55419: BlockingRecv error: recv got EOF from 127.2.193.129:55419 (error 108)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":125,"threads_started":1}
W20260924 10:35:29.122483 11732 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:46035 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.193.190:46035: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:29.128705 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.122931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:42847)
0924 10:35:29.123140 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.123156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.123252 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.123531 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.123534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.123706 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.123832 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.123839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.124583 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.124586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.125550 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.125563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.126023 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.126029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.126123 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.126622 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.126637 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.128596 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130}
W20260924 10:35:29.128906 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.123049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:42847 (local address 127.2.193.190:46035)
0924 10:35:29.123150 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.123153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.123223 (+ 70us) server_negotiation.cc:407] Connection header received
0924 10:35:29.123386 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.123388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.123431 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.123491 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.123943 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.124481 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.125683 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.125907 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.126002 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.128780 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:42847: BlockingRecv error: recv got EOF from 127.2.193.129:42847 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:29.134528 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.129370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:55991)
0924 10:35:29.129580 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.129592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.129653 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.129895 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.129897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.130021 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.130094 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.130098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.131071 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.131074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.131593 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.131600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.132105 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.132107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.132163 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.132639 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.132650 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.134413 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146}
W20260924 10:35:29.134682 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.129473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:55991 (local address 127.2.193.190:46035)
0924 10:35:29.129604 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.129609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.129640 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:35:29.129739 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.129743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.129794 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.129873 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.130192 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.130966 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.131702 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.131999 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.132098 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.134569 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:55991: BlockingRecv error: recv got EOF from 127.2.193.129:55991 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 10:35:29.134775 11732 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:35:29.227032 11734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:29.227403 11734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:29.227496 11734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:29.231655 11734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:29.231778 11734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:29.231889 11734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:29.237614 11734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.130.keytab
--principal=oryx/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:45605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=41383
--tserver_master_addrs=127.2.193.190:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:29.238786 11734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:29.239866 11734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:29.250094 11734 server_base.cc:1034] running on GCE node
W20260924 10:35:29.250056 11742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:29.250052 11741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:29.250052 11744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:29.250780 11734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:29.251464 11734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:29.252652 11734 hybrid_clock.cc:648] HybridClock initialized: now 1790246129252633 us; error 32 us; skew 500 ppm
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246129, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:29.256217 11734 init.cc:377] Logged in from keytab as oryx/127.2.193.130@KRBTEST.COM (short username oryx)
I20260924 10:35:29.257506 11734 webserver.cc:460] Webserver started at http://127.2.193.130:41383/ using document root <none> and password file <none>
I20260924 10:35:29.258132 11734 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:29.258206 11734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:29.262166 11734 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:35:29.264221 11751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.265503 11734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:29.265627 11734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "85cc30395de64ce995fc9aa53ef0c902"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "c0de6e0024469a248b6d8632bd9bfec1"
server_key_iv: "e4cc9e93a21d58cb0554d8c0d06055a9"
server_key_version: "encryptionkey@0"
I20260924 10:35:29.266170 11734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:29.297091 11734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:29.300603 11734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:29.300805 11734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:29.301477 11734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:29.302356 11734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:29.302423 11734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.302485 11734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:29.302524 11734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.311534 11734 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:45605
I20260924 10:35:29.311561 11864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:45605 every 8 connection(s)
I20260924 10:35:29.312547 11734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:29.316676 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11734
I20260924 10:35:29.317102 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:36245
--local_ip_for_outbound_sockets=127.2.193.131
--tserver_master_addrs=127.2.193.190:46035
--webserver_port=43185
--webserver_interface=127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:35:29.324705 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.314299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:52095 (local address 127.2.193.190:46035)
0924 10:35:29.314440 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.314444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.315452 (+ 1008us) server_negotiation.cc:407] Connection header received
0924 10:35:29.316272 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.316275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.316323 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.316398 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.317767 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.318619 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.320390 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.320597 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.320672 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.324581 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:52095: BlockingRecv error: recv got EOF from 127.2.193.130:52095 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:35:29.325395 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.314652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:52095)
0924 10:35:29.315303 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.315345 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.316098 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.316580 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.316586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.317025 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.317611 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.317625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.318748 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.318753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.320263 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.320277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.320754 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.320761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.321014 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.321661 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.321688 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.324457 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":120,"threads_started":1}
W20260924 10:35:29.326045 11865 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:46035 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.193.190:46035: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:29.332001 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.326487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:48847)
0924 10:35:29.326597 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.326614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.326691 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.327113 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.327115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.327240 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.327365 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.327371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.328479 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.328482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.329015 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.329022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.329589 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.329591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.329660 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.330126 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.330138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.331913 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:35:29.332186 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.326683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:48847 (local address 127.2.193.190:46035)
0924 10:35:29.326837 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.326843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.326862 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:35:29.326916 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.326921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.326987 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.327082 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.327480 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.328367 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.329121 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.329486 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.329591 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.332072 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:48847: BlockingRecv error: recv got EOF from 127.2.193.130:48847 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:29.337800 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.332722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:44139)
0924 10:35:29.332830 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.332847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.332921 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.333153 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.333155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.333318 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.333407 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.333411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.334400 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.334403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.334929 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.334936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.335511 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.335514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.335571 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.336031 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.336042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.337706 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:35:29.337988 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.332803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:44139 (local address 127.2.193.190:46035)
0924 10:35:29.332933 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.332937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.332980 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:35:29.333015 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.333018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.333058 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.333122 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.333510 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.334298 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.335038 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.335399 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.335494 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.337854 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:44139: BlockingRecv error: recv got EOF from 127.2.193.130:44139 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:35:29.338090 11865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:35:29.432060 11868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:29.432356 11868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:29.432427 11868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:29.436534 11868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:29.436630 11868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:29.436720 11868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:29.442040 11868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/krb5kdc/oryx_127.2.193.131.keytab
--principal=oryx/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:36245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=43185
--tserver_master_addrs=127.2.193.190:46035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:29.443096 11868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:29.444129 11868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:29.454039 11876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:29.454023 11874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:29.454006 11873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:29.454269 11868 server_base.cc:1034] running on GCE node
I20260924 10:35:29.454841 11868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:29.455451 11868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:29.456633 11868 hybrid_clock.cc:648] HybridClock initialized: now 1790246129456621 us; error 31 us; skew 500 ppm
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246129, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:29.459371 11868 init.cc:377] Logged in from keytab as oryx/127.2.193.131@KRBTEST.COM (short username oryx)
I20260924 10:35:29.460179 11868 webserver.cc:460] Webserver started at http://127.2.193.131:43185/ using document root <none> and password file <none>
I20260924 10:35:29.460659 11868 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:29.460723 11868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:29.464325 11868 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:29.466216 11883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.467371 11868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:29.467458 11868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "e41a0239c2e04ff49336554dfc1665e2"
format_stamp: "Formatted at 2026-09-24 10:35:28 on dist-test-slave-8wc7"
server_key: "22bf61b1c46f80f15351de206ab501e3"
server_key_iv: "0adbf0aedf69f66036dd84d64d874518"
server_key_version: "encryptionkey@0"
I20260924 10:35:29.467981 11868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:29.480026 11868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:29.483162 11868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:29.483304 11868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:29.483891 11868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:29.484647 11868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:29.484689 11868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.484755 11868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:29.484787 11868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:29.492937 11868 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:36245
I20260924 10:35:29.493011 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:36245 every 8 connection(s)
I20260924 10:35:29.494035 11868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:35:29.503984 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.495708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:57365 (local address 127.2.193.190:46035)
0924 10:35:29.495848 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.495852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.496766 (+ 914us) server_negotiation.cc:407] Connection header received
0924 10:35:29.497714 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.497717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.497775 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.497876 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.498946 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.499456 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.500549 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.500767 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.500872 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.503855 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:57365: BlockingRecv error: recv got EOF from 127.2.193.131:57365 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:35:29.504014 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 11868
I20260924 10:35:29.504786 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.496030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:57365)
0924 10:35:29.496625 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.496663 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.497514 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.498044 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.498051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.498367 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.498803 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.498815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.499575 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.499578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.500418 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.500427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.500904 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.500908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.501070 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.501622 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.501637 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.503749 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":107,"threads_started":1}
W20260924 10:35:29.505296 11997 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:46035 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.193.190:46035: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:29.510741 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.505688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:57117)
0924 10:35:29.505788 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.505802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.505867 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.506227 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.506230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.506375 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.506452 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.506457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.507145 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.507152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.507681 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.507688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.508111 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.508115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.508200 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.508744 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.508759 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.510595 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:35:29.510835 12001 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.505798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:57117 (local address 127.2.193.190:46035)
0924 10:35:29.506053 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.506056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.506066 (+ 10us) server_negotiation.cc:407] Connection header received
0924 10:35:29.506091 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.506092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.506121 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.506215 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.506557 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.507046 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.507791 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.508011 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.508112 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.510713 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:57117: BlockingRecv error: recv got EOF from 127.2.193.131:57117 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20260924 10:35:29.513792 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.504957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.2.193.190:46035)
0924 10:35:29.505077 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.505081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.505094 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:35:29.505268 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.505271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.505317 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.505404 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.506310 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.506796 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.507656 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.507864 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.507969 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.509428 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:29.509434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:29.509475 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:29.509510 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:29.511543 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:29.512164 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:29.512169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:29.512174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:29.512325 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:29.512584 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:29.512587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:29.512589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:29.513004 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:29.513152 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:29.513449 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 10:35:29.513674 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:29 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:29.516656 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:29.511444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:38757)
0924 10:35:29.511605 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:29.511620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:29.511677 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:29.511896 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:29.511900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:29.512047 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:29.512149 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.512157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.513033 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.513035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.513769 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:29.513778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.514278 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.514281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:29.514337 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.514776 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:29.514790 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:29.516529 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 10:35:29.516731 12001 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:29.511515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:38757 (local address 127.2.193.190:46035)
0924 10:35:29.511636 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:29.511638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:29.511650 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:35:29.511756 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:29.511758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:29.511787 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:29.511849 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:29.512259 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.512922 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.513875 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:29.514166 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:29.514259 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:29.516629 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:38757: BlockingRecv error: recv got EOF from 127.2.193.131:38757 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 10:35:29.516845 11997 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 10:35:30 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:30 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:30.140992 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:30.135409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:45423)
0924 10:35:30.135720 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:30.135734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:30.135825 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:30.136058 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:30.136059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:30.136237 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:30.136337 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:30.136342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.137068 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.137070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:30.137720 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:30.137731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.138149 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.138152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:30.138212 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:30.138885 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:30.138897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:30.140855 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
W20260924 10:35:30.141243 12003 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:30.135525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:45423 (local address 127.2.193.190:46035)
0924 10:35:30.135790 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:30.135794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:30.135808 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:30.135919 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:30.135921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:30.135967 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:30.136045 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:30.136437 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.136972 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.137833 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.138053 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.138130 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:30.141025 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:45423: BlockingRecv error: recv got EOF from 127.2.193.129:45423 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
Sep 24 10:35:30 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:30 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:30.344161 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:30.338767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:57827)
0924 10:35:30.339079 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:30.339093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:30.339195 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:30.339439 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:30.339441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:30.339634 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:30.339728 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:30.339734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.340440 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.340442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:30.341011 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:30.341018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.341492 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.341495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:30.341556 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:30.342122 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:30.342134 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:30.344031 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":85,"threads_started":1}
W20260924 10:35:30.344352 12003 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:30.338859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:57827 (local address 127.2.193.190:46035)
0924 10:35:30.338993 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:30.338997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:30.339169 (+ 172us) server_negotiation.cc:407] Connection header received
0924 10:35:30.339276 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:30.339278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:30.339324 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:30.339398 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:30.339828 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.340346 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.341107 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.341388 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.341472 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:30.344205 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:57827: BlockingRecv error: recv got EOF from 127.2.193.130:57827 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 10:35:30 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:30 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:30.522915 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:30.517529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:48029)
0924 10:35:30.517822 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:30.517836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:30.517936 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:30.518175 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:30.518177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:30.518349 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:30.518479 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:30.518485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.519184 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.519187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:30.519763 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:30.519771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.520182 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.520184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:30.520253 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:30.520742 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:30.520754 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:30.522760 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
W20260924 10:35:30.523052 12003 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:30.517645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:48029 (local address 127.2.193.190:46035)
0924 10:35:30.517788 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:30.517792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:30.517906 (+ 114us) server_negotiation.cc:407] Connection header received
0924 10:35:30.518018 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:30.518020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:30.518065 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:30.518138 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:30.518584 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.519093 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.519868 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:30.520084 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:30.520163 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:30.522915 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:48029: BlockingRecv error: recv got EOF from 127.2.193.131:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:35:31 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:31 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:31.147895 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:31.141886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:36527)
0924 10:35:31.142183 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:31.142198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:31.142290 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:31.142685 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:31.142687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:31.142879 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:31.142988 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:31.142994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.143716 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.143719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:31.144366 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:31.144377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.144988 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.144992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:31.145086 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:31.145744 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:31.145756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:31.147701 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
W20260924 10:35:31.147951 12007 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:31.142014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:36527 (local address 127.2.193.190:46035)
0924 10:35:31.142256 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:31.142259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:31.142273 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:31.142471 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:31.142474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:31.142533 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:31.142619 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:31.143099 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.143617 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.144494 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.144863 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.144988 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:31.147830 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:36527: BlockingRecv error: recv got EOF from 127.2.193.129:36527 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":61,"threads_started":1}
Sep 24 10:35:31 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:31 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:31.351083 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:31.345328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:50503)
0924 10:35:31.345622 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:31.345639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:31.345769 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:31.346056 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:31.346059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:31.346265 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:31.346353 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:31.346359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.347195 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.347202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:31.347883 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:31.347892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.348352 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.348356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:31.348434 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:31.349014 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:31.349026 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:31.350882 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260924 10:35:31.351121 12007 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:31.345429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:50503 (local address 127.2.193.190:46035)
0924 10:35:31.345609 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:31.345612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:31.345731 (+ 119us) server_negotiation.cc:407] Connection header received
0924 10:35:31.345891 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:31.345896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:31.345945 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:31.346041 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:31.346479 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.347085 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.348007 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.348238 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.348350 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:31.351016 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:50503: BlockingRecv error: recv got EOF from 127.2.193.130:50503 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 10:35:31 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:31 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:31.529835 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:31.523884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:32913)
0924 10:35:31.524208 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:31.524225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:31.524355 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:31.524641 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:31.524644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:31.524883 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:31.525020 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:31.525026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.525998 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.526001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:31.526592 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:31.526600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.527075 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.527077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:31.527141 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:31.527758 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:31.527772 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:31.529707 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
W20260924 10:35:31.530074 12007 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:31.524099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:32913 (local address 127.2.193.190:46035)
0924 10:35:31.524280 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:31.524285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:31.524301 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:35:31.524459 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:31.524461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:31.524507 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:31.524578 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:31.525312 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.525873 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.526733 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:31.526967 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:31.527074 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:31.529944 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:32913: BlockingRecv error: recv got EOF from 127.2.193.131:32913 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 10:35:32 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:32 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:32.154009 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:32.148644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:39495)
0924 10:35:32.148910 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:32.148925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:32.149031 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:32.149299 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:32.149302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:32.149512 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:32.149615 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:32.149621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.150340 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.150343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:32.150887 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:32.150894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.151299 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.151301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:32.151363 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:32.151874 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:32.151885 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:32.153830 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260924 10:35:32.154093 12011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:32.148697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:39495 (local address 127.2.193.190:46035)
0924 10:35:32.148929 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:32.148933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:32.148984 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:35:32.149098 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:32.149100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:32.149142 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:32.149271 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:32.149723 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.150249 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.150990 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.151204 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.151277 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:32.153957 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:39495: BlockingRecv error: recv got EOF from 127.2.193.129:39495 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":58,"threads_started":1}
Sep 24 10:35:32 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:32 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:32.356925 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:32.351882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:50439)
0924 10:35:32.352132 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:32.352145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:32.352237 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:32.352458 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:32.352460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:32.352638 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:32.352731 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:32.352736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.353462 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.353464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:32.354022 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:32.354030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.354418 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.354421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:32.354482 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:32.354974 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:32.354985 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:32.356808 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
W20260924 10:35:32.357070 12011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:32.351960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:50439 (local address 127.2.193.190:46035)
0924 10:35:32.352092 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:32.352100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:32.352210 (+ 110us) server_negotiation.cc:407] Connection header received
0924 10:35:32.352313 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:32.352315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:32.352356 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:32.352423 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:32.352825 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.353367 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.354121 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.354327 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.354399 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:32.356951 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:50439: BlockingRecv error: recv got EOF from 127.2.193.130:50439 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 10:35:32 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:32 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:32.536062 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:32.530637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:51173)
0924 10:35:32.530904 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:32.530918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:32.531022 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:32.531251 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:32.531253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:32.531441 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:32.531538 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:32.531544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.532255 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.532257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:32.532810 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:32.532816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.533250 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.533252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:32.533313 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:32.533801 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:32.533812 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:32.535878 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
W20260924 10:35:32.536291 12011 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:32.530733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:51173 (local address 127.2.193.190:46035)
0924 10:35:32.530866 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:32.530877 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:32.530997 (+ 120us) server_negotiation.cc:407] Connection header received
0924 10:35:32.531100 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:32.531102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:32.531145 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:32.531214 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:32.531636 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.532154 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.532904 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:32.533116 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:32.533259 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:32.536026 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:51173: BlockingRecv error: recv got EOF from 127.2.193.131:51173 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 10:35:33 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:33 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:33.164556 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:33.154945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:36599)
0924 10:35:33.155227 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:33.155242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:33.155342 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:33.155640 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:33.155643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:33.155819 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:33.155918 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:33.155923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.156669 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.156671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:33.157330 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:33.157339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.157814 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.157817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:33.157896 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:33.162376 (+ 4480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:33.162392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:33.164390 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
W20260924 10:35:33.164687 12015 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:33.155008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:36599 (local address 127.2.193.190:46035)
0924 10:35:33.155249 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:33.155253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:33.155296 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:35:33.155421 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:33.155423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:33.155475 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:33.155565 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:33.156031 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.156567 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.157462 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.157687 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.157765 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:33.164557 (+ 6792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:36599: BlockingRecv error: recv got EOF from 127.2.193.129:36599 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
Sep 24 10:35:33 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:33 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:33.363340 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:33.357945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:44595)
0924 10:35:33.358222 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:33.358237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:33.358343 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:33.358598 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:33.358600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:33.358780 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:33.358877 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:33.358883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.359668 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.359670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:33.360219 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:33.360226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.360648 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.360650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:33.360713 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:33.361291 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:33.361303 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:33.363191 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
W20260924 10:35:33.363498 12015 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:33.358090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:44595 (local address 127.2.193.190:46035)
0924 10:35:33.358249 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:33.358253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:33.358313 (+ 60us) server_negotiation.cc:407] Connection header received
0924 10:35:33.358443 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:33.358445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:33.358488 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:33.358561 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:33.358992 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.359557 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.360321 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.360553 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.360630 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:33.363344 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:44595: BlockingRecv error: recv got EOF from 127.2.193.130:44595 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 10:35:33 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:33 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:33.542109 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:33.536870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:40245)
0924 10:35:33.537133 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:33.537148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:33.537278 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:33.537508 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:33.537511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:33.537683 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:33.537776 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:33.537782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.538488 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.538490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:33.539044 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:33.539051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.539441 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.539444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:33.539504 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:33.539991 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:33.540003 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:33.541987 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
W20260924 10:35:33.542255 12015 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:33.536940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:40245 (local address 127.2.193.190:46035)
0924 10:35:33.537083 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:33.537086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:33.537250 (+ 164us) server_negotiation.cc:407] Connection header received
0924 10:35:33.537357 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:33.537359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:33.537404 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:33.537486 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:33.537884 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.538394 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.539142 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:33.539347 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:33.539420 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:33.542129 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:40245: BlockingRecv error: recv got EOF from 127.2.193.131:40245 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 10:35:34 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:34 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:34.171927 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:34.165599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:57713)
0924 10:35:34.165875 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:34.165898 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:34.165987 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:34.166232 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:34.166234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:34.166418 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:34.166512 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:34.166518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.167576 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.167580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:34.168387 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:34.168397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.168858 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.168861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:34.168942 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:34.169618 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:34.169640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:34.171739 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260924 10:35:34.172043 12019 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:34.165692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:57713 (local address 127.2.193.190:46035)
0924 10:35:34.165958 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:34.165961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:34.165979 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:35:34.166064 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:34.166065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:34.166106 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:34.166198 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:34.166703 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.167452 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.168513 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.168745 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.168832 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:34.171896 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:57713: BlockingRecv error: recv got EOF from 127.2.193.129:57713 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
Sep 24 10:35:34 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:34 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:34.370162 12020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:34.364256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:45933)
0924 10:35:34.364532 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:34.364546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:34.364629 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:34.364867 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:34.364870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:34.365057 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:34.365147 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:34.365153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.365957 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.365962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:34.366527 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:34.366535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.367005 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.367009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:34.367093 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:34.367678 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:34.367690 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:34.369973 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
W20260924 10:35:34.370352 12019 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:34.364427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:45933 (local address 127.2.193.190:46035)
0924 10:35:34.364566 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:34.364570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:34.364586 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:35:34.364716 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:34.364718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:34.364759 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:34.364834 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:34.365302 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.365840 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.366651 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.366883 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.367003 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:34.370192 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:45933: BlockingRecv error: recv got EOF from 127.2.193.130:45933 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 10:35:34 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:34 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:34.549275 12021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:34.543146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:49803)
0924 10:35:34.543467 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:34.543480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:34.543614 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:34.543894 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:34.543897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:34.544113 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:34.544203 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:34.544209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.545002 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.545005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:34.545611 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:34.545620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.546135 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.546139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:34.546224 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:34.546849 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:34.546862 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:34.549077 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
W20260924 10:35:34.549417 12019 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:34.543225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:49803 (local address 127.2.193.190:46035)
0924 10:35:34.543384 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:34.543389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:34.543564 (+ 175us) server_negotiation.cc:407] Connection header received
0924 10:35:34.543716 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:34.543719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:34.543774 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:34.543872 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:34.544334 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.544872 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.545734 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:34.546018 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:34.546132 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:34.549282 (+ 3150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:49803: BlockingRecv error: recv got EOF from 127.2.193.131:49803 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 10:35:35 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:35 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:35.179265 12022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:35.172984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:51751)
0924 10:35:35.173501 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:35.173720 (+ 219us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:35.173843 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:35.174146 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:35.174148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:35.174333 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:35.174427 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:35.174433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.175281 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.175283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:35.175832 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:35.175839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.176303 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.176306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:35.176364 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:35.176909 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:35.176923 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:35.179147 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":74,"threads_started":1}
W20260924 10:35:35.179450 12023 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:35.173016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:51751 (local address 127.2.193.190:46035)
0924 10:35:35.173343 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:35.173346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:35.173795 (+ 449us) server_negotiation.cc:407] Connection header received
0924 10:35:35.173922 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:35.173924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:35.173977 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:35.174071 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:35.174589 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.175179 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.175949 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.176198 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.176310 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:35.179276 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:51751: BlockingRecv error: recv got EOF from 127.2.193.129:51751 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
Sep 24 10:35:35 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:35 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:35.377234 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:35.371191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:59989)
0924 10:35:35.371517 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:35.371539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:35.371641 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:35.371880 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:35.371882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:35.372141 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:35.372306 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:35.372313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.373366 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.373370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:35.373968 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:35.373977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.374494 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.374497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:35.374585 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:35.375157 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:35.375169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:35.377024 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
W20260924 10:35:35.377322 12023 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:35.371265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:59989 (local address 127.2.193.190:46035)
0924 10:35:35.371454 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:35.371459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:35.371633 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:35:35.371726 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:35.371728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:35.371778 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:35.371839 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:35.372476 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.373159 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.374106 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.374373 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.374488 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:35.377159 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:59989: BlockingRecv error: recv got EOF from 127.2.193.130:59989 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 10:35:35 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:35 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:35.555646 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:35.550089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:56373)
0924 10:35:35.550385 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:35.550399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:35.550542 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:35.550809 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:35.550812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:35.551038 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:35.551130 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:35.551136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.551899 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.551906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:35.552494 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:35.552504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.552965 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.552968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:35.553054 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:35.553652 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:35.553664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:35.555472 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
W20260924 10:35:35.555724 12023 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:35.550158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:56373 (local address 127.2.193.190:46035)
0924 10:35:35.550321 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:35.550325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:35.550478 (+ 153us) server_negotiation.cc:407] Connection header received
0924 10:35:35.550642 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:35.550646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:35.550698 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:35.550786 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:35.551253 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.551781 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.552622 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:35.552850 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:35.552961 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:35.555603 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:56373: BlockingRecv error: recv got EOF from 127.2.193.131:56373 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 10:35:36 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:36 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:36.185240 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:36.179988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:44563)
0924 10:35:36.180284 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:36.180298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:36.180397 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:36.180643 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:36.180645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:36.180826 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:36.180930 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:36.180936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.181674 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.181676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:36.182236 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:36.182244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.182662 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.182664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:36.182725 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:36.183201 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:36.183212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:36.185018 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
W20260924 10:35:36.185299 12027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:36.180046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:44563 (local address 127.2.193.190:46035)
0924 10:35:36.180306 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:36.180309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:36.180352 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:35:36.180483 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:36.180485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:36.180528 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:36.180604 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:36.181029 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.181583 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.182337 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.182564 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.182637 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:36.185140 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:44563: BlockingRecv error: recv got EOF from 127.2.193.129:44563 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":116,"threads_started":1}
Sep 24 10:35:36 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:36 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:36.383323 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:36.378052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:38193)
0924 10:35:36.378408 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:36.378425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:36.378540 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:36.378844 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:36.378847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:36.379031 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:36.379124 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:36.379129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.379878 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.379880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:36.380453 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:36.380461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.380875 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.380877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:36.380938 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:36.381429 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:36.381441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:36.383185 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
W20260924 10:35:36.383412 12027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:36.378111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:38193 (local address 127.2.193.190:46035)
0924 10:35:36.378270 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:36.378274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:36.378513 (+ 239us) server_negotiation.cc:407] Connection header received
0924 10:35:36.378632 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:36.378635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:36.378715 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:36.378800 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:36.379224 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.379771 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.380557 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.380775 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.380847 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:36.383320 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:38193: BlockingRecv error: recv got EOF from 127.2.193.130:38193 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 10:35:36 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:36 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:36.561868 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:36.556541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:59223)
0924 10:35:36.556833 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:36.556848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:36.556953 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:36.557169 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:36.557218 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:36.557411 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:36.557507 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:36.557514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.558236 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.558240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:36.558812 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:36.558829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.559248 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.559252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:36.559321 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:36.559818 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:36.559830 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:36.561745 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":69,"threads_started":1}
W20260924 10:35:36.562009 12027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:36.556647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:59223 (local address 127.2.193.190:46035)
0924 10:35:36.556820 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:36.556824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:36.556918 (+ 94us) server_negotiation.cc:407] Connection header received
0924 10:35:36.557020 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:36.557022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:36.557066 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:36.557131 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:36.557608 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.558131 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.558922 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:36.559130 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:36.559205 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:36.561885 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:59223: BlockingRecv error: recv got EOF from 127.2.193.131:59223 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 10:35:37 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:37 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:37.191387 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:37.186050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:34809)
0924 10:35:37.186313 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:37.186327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:37.186425 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:37.186679 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:37.186681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:37.186868 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:37.186962 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:37.186967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.187723 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.187726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:37.188324 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:37.188334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.188748 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.188751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:37.188844 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:37.189392 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:37.189404 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:37.191242 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
W20260924 10:35:37.191643 12031 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:37.186223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:34809 (local address 127.2.193.190:46035)
0924 10:35:37.186472 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:37.186477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:37.186490 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:35:37.186525 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:37.186527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:37.186566 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:37.186644 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:37.187060 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.187599 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.188432 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.188648 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.188720 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:37.191387 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:34809: BlockingRecv error: recv got EOF from 127.2.193.129:34809 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
Sep 24 10:35:37 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:37 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:37.389732 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:37.384206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:49201)
0924 10:35:37.384520 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:37.384538 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:37.384658 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:37.384942 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:37.384945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:37.385219 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:37.385342 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:37.385350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.386211 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.386214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:37.386766 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:37.386772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.387172 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.387174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:37.387234 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:37.387738 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:37.387751 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:37.389594 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
W20260924 10:35:37.389986 12031 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:37.384296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:49201 (local address 127.2.193.190:46035)
0924 10:35:37.384450 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:37.384454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:37.384620 (+ 166us) server_negotiation.cc:407] Connection header received
0924 10:35:37.384753 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:37.384756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:37.384816 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:37.384898 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:37.385457 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.386118 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.386865 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.387077 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.387152 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:37.389754 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:49201: BlockingRecv error: recv got EOF from 127.2.193.130:49201 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 10:35:37 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:37 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:37.568068 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:37.562759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:41885)
0924 10:35:37.563045 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:37.563060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:37.563193 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:37.563456 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:37.563458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:37.563671 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:37.563762 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:37.563767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.564510 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.564515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:37.565077 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:37.565084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.565556 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.565560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:37.565645 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:37.566133 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:37.566145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:37.567897 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
W20260924 10:35:37.568156 12031 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:37.562827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:41885 (local address 127.2.193.190:46035)
0924 10:35:37.563000 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:37.563005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:37.563139 (+ 134us) server_negotiation.cc:407] Connection header received
0924 10:35:37.563292 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:37.563296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:37.563350 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:37.563435 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:37.563875 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.564399 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.565215 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:37.565442 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:37.565555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:37.568029 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:41885: BlockingRecv error: recv got EOF from 127.2.193.131:41885 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 10:35:38 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:38 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:38.204893 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:38.192201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:47793)
0924 10:35:38.192476 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:38.192491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:38.192589 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:38.192876 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:38.192878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:38.193068 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:38.193199 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:38.193205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.193927 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.193930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:38.194480 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:38.194487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.194890 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.194893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:38.194950 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:38.202698 (+ 7748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:38.202712 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:38.204747 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260924 10:35:38.205060 12035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:38.192419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:47793 (local address 127.2.193.190:46035)
0924 10:35:38.192651 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:38.192655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:38.192668 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:35:38.192706 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:38.192708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:38.192749 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:38.192830 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:38.193288 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.193820 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.194582 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.194797 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.194881 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:38.204919 (+ 10038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:47793: BlockingRecv error: recv got EOF from 127.2.193.129:47793 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
Sep 24 10:35:38 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:38 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:38.395831 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:38.390495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:48459)
0924 10:35:38.390772 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:38.390786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:38.390876 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:38.391124 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:38.391126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:38.391307 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:38.391416 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:38.391422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.392132 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.392134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:38.392699 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:38.392706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.393129 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.393131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:38.393228 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:38.393734 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:38.393746 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:38.395699 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
W20260924 10:35:38.395982 12035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:38.390653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:48459 (local address 127.2.193.190:46035)
0924 10:35:38.390813 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:38.390816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:38.390832 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:35:38.390962 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:38.390964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:38.391014 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:38.391087 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:38.391512 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.392045 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.392801 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.393029 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.393120 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:38.395852 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:48459: BlockingRecv error: recv got EOF from 127.2.193.130:48459 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 10:35:38 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:38 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:38.574644 12037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:38.568841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:55407)
0924 10:35:38.569141 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:38.569165 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:38.569283 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:38.569546 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:38.569548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:38.569739 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:38.569830 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:38.569836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.570690 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.570692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:38.571321 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:38.571330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.571752 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.571758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:38.571824 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:38.572411 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:38.572423 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:38.574523 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":124,"threads_started":1}
W20260924 10:35:38.574821 12035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:38.568968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:55407 (local address 127.2.193.190:46035)
0924 10:35:38.569108 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:38.569111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:38.569249 (+ 138us) server_negotiation.cc:407] Connection header received
0924 10:35:38.569376 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:38.569379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:38.569436 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:38.569538 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:38.569923 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.570572 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.571417 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:38.571636 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:38.571714 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:38.574661 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:55407: BlockingRecv error: recv got EOF from 127.2.193.131:55407 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 10:35:39 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:39 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:39.211730 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:39.205958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:43729)
0924 10:35:39.206235 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:39.206251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:39.206340 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:39.206604 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:39.206606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:39.206791 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:39.206902 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:39.206908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.207686 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.207689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:39.208257 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:39.208266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.208692 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.208695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:39.208760 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:39.209355 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:39.209367 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:39.211557 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
W20260924 10:35:39.211807 12039 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:39.206086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:43729 (local address 127.2.193.190:46035)
0924 10:35:39.206316 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:39.206320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:39.206335 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:35:39.206428 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:39.206431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:39.206475 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:39.206565 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:39.207020 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.207567 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.208361 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.208578 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.208652 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:39.211684 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:43729: BlockingRecv error: recv got EOF from 127.2.193.129:43729 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
Sep 24 10:35:39 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:39 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:39.401862 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:39.396726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:41585)
0924 10:35:39.396975 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:39.396988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:39.397077 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:39.397332 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:39.397334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:39.397515 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:39.397609 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:39.397614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.398362 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.398365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:39.398925 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:39.398933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.399338 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.399341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:39.399404 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:39.399895 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:39.399905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:39.401745 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
W20260924 10:35:39.402093 12039 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:39.396814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:41585 (local address 127.2.193.190:46035)
0924 10:35:39.397004 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:39.397007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:39.397039 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:35:39.397162 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:39.397164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:39.397230 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:39.397300 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:39.397727 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.398240 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.399040 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.399245 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.399321 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:39.401891 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:41585: BlockingRecv error: recv got EOF from 127.2.193.130:41585 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 10:35:39 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:39 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:39.580675 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:39.575402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:40675)
0924 10:35:39.575678 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:39.575692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:39.575795 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:39.576035 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:39.576037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:39.576215 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:39.576309 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:39.576314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.577051 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.577053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:39.577654 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:39.577663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.578067 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.578070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:39.578129 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:39.578640 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:39.578651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:39.580537 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260924 10:35:39.580905 12039 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:39.575557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:40675 (local address 127.2.193.190:46035)
0924 10:35:39.575691 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:39.575695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:39.575770 (+ 75us) server_negotiation.cc:407] Connection header received
0924 10:35:39.575877 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:39.575879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:39.575923 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:39.576004 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:39.576434 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.576946 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.577753 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:39.577965 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:39.578047 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:39.580722 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:40675: BlockingRecv error: recv got EOF from 127.2.193.131:40675 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 10:35:40 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:40 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:40.218583 12042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:40.212640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:35731)
0924 10:35:40.212933 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:40.212950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:40.213071 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:40.213426 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:40.213429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:40.213679 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:40.213809 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:40.213819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.214592 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.214594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:40.215160 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:40.215169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.215625 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.215628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:40.215696 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:40.216220 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:40.216232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:40.218440 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
W20260924 10:35:40.218827 12043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:40.212692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:35731 (local address 127.2.193.190:46035)
0924 10:35:40.212940 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:40.212945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:40.213019 (+ 74us) server_negotiation.cc:407] Connection header received
0924 10:35:40.213201 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:40.213204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:40.213264 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:40.213382 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:40.213929 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.214471 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.215263 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.215493 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.215572 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:40.218667 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:35731: BlockingRecv error: recv got EOF from 127.2.193.129:35731 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":59,"threads_started":1}
Sep 24 10:35:40 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:40 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:40.408452 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:40.402801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:48885)
0924 10:35:40.403129 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:40.403145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:40.403267 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:40.403615 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:40.403619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:40.403856 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:40.403977 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:40.403984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.404924 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.404927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:40.405506 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:40.405514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.405918 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.405920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:40.405982 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:40.406486 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:40.406497 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:40.408333 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
W20260924 10:35:40.408653 12043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:40.402857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:48885 (local address 127.2.193.190:46035)
0924 10:35:40.403012 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:40.403017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:40.403257 (+ 240us) server_negotiation.cc:407] Connection header received
0924 10:35:40.403396 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:40.403399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:40.403462 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:40.403564 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:40.404092 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.404821 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.405604 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.405813 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.405899 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:40.408486 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:48885: BlockingRecv error: recv got EOF from 127.2.193.130:48885 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 10:35:40 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:40 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:40.586609 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:40.581456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:42441)
0924 10:35:40.581716 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:40.581732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:40.581837 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:40.582055 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:40.582057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:40.582227 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:40.582320 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:40.582325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.583049 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.583052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:40.583615 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:40.583623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.584014 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.584017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:40.584074 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:40.584550 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:40.584561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:40.586494 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260924 10:35:40.586742 12043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:40.581536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:42441 (local address 127.2.193.190:46035)
0924 10:35:40.581675 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:40.581679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:40.581791 (+ 112us) server_negotiation.cc:407] Connection header received
0924 10:35:40.581915 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:40.581917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:40.581962 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:40.582021 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:40.582419 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.582949 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.583713 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:40.583923 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:40.583995 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:40.586625 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:42441: BlockingRecv error: recv got EOF from 127.2.193.131:42441 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 10:35:41 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:41 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:41.225437 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:41.219540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:55107)
0924 10:35:41.219832 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:41.219848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:41.219944 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:41.220239 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:41.220241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:41.220453 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:41.220550 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:41.220556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.221365 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.221368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:41.221943 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:41.221952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.222397 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.222400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:41.222470 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:41.222997 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:41.223020 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:41.225295 (+ 2275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
W20260924 10:35:41.225589 12047 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:41.219726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:55107 (local address 127.2.193.190:46035)
0924 10:35:41.220008 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:41.220011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:41.220026 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:35:41.220068 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:41.220070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:41.220111 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:41.220201 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:41.220670 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.221228 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.222060 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.222283 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.222362 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:41.225427 (+ 3065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:55107: BlockingRecv error: recv got EOF from 127.2.193.129:55107 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
Sep 24 10:35:41 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:41 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:41.414764 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:41.409274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:51449)
0924 10:35:41.409587 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:41.409601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:41.409700 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:41.409921 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:41.409924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:41.410105 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:41.410198 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:41.410204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.410964 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.410968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:41.411685 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:41.411695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.412199 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.412204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:41.412311 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:41.412817 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:41.412829 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:41.414616 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":114,"threads_started":1}
W20260924 10:35:41.414897 12047 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:41.409433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:51449 (local address 127.2.193.190:46035)
0924 10:35:41.409591 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:41.409594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:41.409650 (+ 56us) server_negotiation.cc:407] Connection header received
0924 10:35:41.409764 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:41.409766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:41.409808 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:41.409884 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:41.410296 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.410816 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.411806 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.412109 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.412186 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:41.414785 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:51449: BlockingRecv error: recv got EOF from 127.2.193.130:51449 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 10:35:41 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:41 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:41.593542 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:41.587499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:45293)
0924 10:35:41.587848 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:41.587864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:41.588025 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:41.588317 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:41.588320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:41.588532 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:41.588625 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:41.588631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.589544 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.589550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:41.590272 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:41.590281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.590731 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.590734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:41.590798 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:41.591373 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:41.591395 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:41.593420 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
W20260924 10:35:41.593732 12047 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:41.587636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:45293 (local address 127.2.193.190:46035)
0924 10:35:41.587804 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:41.587809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:41.587958 (+ 149us) server_negotiation.cc:407] Connection header received
0924 10:35:41.588142 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:41.588145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:41.588201 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:41.588295 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:41.588748 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.589429 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.590413 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:41.590632 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:41.590717 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:41.593572 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:45293: BlockingRecv error: recv got EOF from 127.2.193.131:45293 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 10:35:42 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:42 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:42.232080 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:42.226329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:49423)
0924 10:35:42.226634 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:42.226648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:42.226747 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:42.227020 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:42.227022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:42.227204 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:42.227316 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:42.227322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.228074 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.228077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:42.228664 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:42.228671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.229230 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.229237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:42.229314 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:42.229940 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:42.229952 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:42.231902 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":124,"threads_started":1}
W20260924 10:35:42.232209 12051 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:42.226447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:49423 (local address 127.2.193.190:46035)
0924 10:35:42.226682 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:42.226686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:42.226700 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:42.226848 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:42.226850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:42.226894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:42.226980 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:42.227424 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.227966 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.228813 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.229052 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.229147 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:42.232075 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:49423: BlockingRecv error: recv got EOF from 127.2.193.129:49423 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
Sep 24 10:35:42 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:42 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:35:42.421346 12051 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:42.415796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:36807 (local address 127.2.193.190:46035)
0924 10:35:42.415971 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:42.415975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:42.415992 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:35:42.416117 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:42.416119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:42.416173 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:42.416251 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:42.416675 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.417209 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.417954 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.418164 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.418239 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:42.421223 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:36807: BlockingRecv error: recv got EOF from 127.2.193.130:36807 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:35:42.421337 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:42.415559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:36807)
0924 10:35:42.415897 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:42.415921 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:42.416014 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:42.416291 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:42.416293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:42.416482 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:42.416575 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:42.416581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.417282 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.417284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:42.417854 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:42.417861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.418260 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.418262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:42.418322 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:42.418891 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:42.418905 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:42.421073 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
Sep 24 10:35:42 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:42 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:42.600399 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:42.594437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:60433)
0924 10:35:42.594811 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:42.594827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:42.594984 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:42.595278 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:42.595280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:42.595521 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:42.595613 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:42.595619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.596399 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.596405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:42.596977 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:42.596986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.597545 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.597549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:42.597640 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:42.598235 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:42.598257 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:42.600284 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
W20260924 10:35:42.600545 12051 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:42.594598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:60433 (local address 127.2.193.190:46035)
0924 10:35:42.594758 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:42.594763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:42.594925 (+ 162us) server_negotiation.cc:407] Connection header received
0924 10:35:42.595099 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:42.595103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:42.595158 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:42.595254 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:42.595743 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.596276 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.597104 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:42.597407 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:42.597540 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:42.600424 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:60433: BlockingRecv error: recv got EOF from 127.2.193.131:60433 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 10:35:43 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:43 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:43.242857 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:43.233061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:60695)
0924 10:35:43.233441 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:43.233461 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:43.233569 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:43.233823 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:43.233826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:43.234010 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:43.234105 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:43.234110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.234857 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.234860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:43.235442 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:43.235451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.235874 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.235877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:43.235942 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:43.240301 (+ 4359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:43.240342 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:43.242664 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
W20260924 10:35:43.243108 12055 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:43.233230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:60695 (local address 127.2.193.190:46035)
0924 10:35:43.233554 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:43.233558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:43.233572 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:35:43.233661 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:43.233663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:43.233706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:43.233775 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:43.234211 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.234754 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.235552 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.235773 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.235853 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:43.242843 (+ 6990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:60695: BlockingRecv error: recv got EOF from 127.2.193.129:60695 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
Sep 24 10:35:43 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:43 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:43.427717 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:43.422181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:57093)
0924 10:35:43.422441 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:43.422464 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:43.422564 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:43.422783 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:43.422785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:43.422982 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:43.423076 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:43.423086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.423893 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.423895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:43.424467 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:43.424476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.424949 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.424951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:43.425022 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:43.425714 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:43.425733 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:43.427587 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
W20260924 10:35:43.427879 12055 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:43.422216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:57093 (local address 127.2.193.190:46035)
0924 10:35:43.422405 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:43.422409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:43.422518 (+ 109us) server_negotiation.cc:407] Connection header received
0924 10:35:43.422632 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:43.422634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:43.422677 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:43.422746 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:43.423214 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.423776 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.424609 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.424837 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.424929 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:43.427742 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:57093: BlockingRecv error: recv got EOF from 127.2.193.130:57093 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 24 10:35:43 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:43 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.131@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:43.607337 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:43.601362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.131:47811)
0924 10:35:43.601634 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:43.601648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:43.601752 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:43.602007 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:43.602009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:43.602196 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:43.602297 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:43.602304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.603424 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.603430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:43.604083 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:43.604092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.604592 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.604595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:43.604682 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:43.605278 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:43.605291 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:43.607159 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
W20260924 10:35:43.607407 12055 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:43.601503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:47811 (local address 127.2.193.190:46035)
0924 10:35:43.601665 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:43.601669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:43.601719 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:35:43.601846 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:43.601848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:43.601895 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:43.601963 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:43.602479 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.603320 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.604221 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:43.604467 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:43.604583 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:43.607308 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.131:47811: BlockingRecv error: recv got EOF from 127.2.193.131:47811 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 10:35:44 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:44 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.129@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:44.251408 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:44.243928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.129:53277)
0924 10:35:44.244288 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:44.244308 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:44.244431 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:44.244771 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:44.244775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:44.245029 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:44.245201 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:44.245214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.246342 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.246347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:44.247425 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:44.247437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.248080 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.248085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:44.248182 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:44.248917 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:44.248935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:44.251232 (+ 2297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
W20260924 10:35:44.251734 12059 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:44.244002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:53277 (local address 127.2.193.190:46035)
0924 10:35:44.244275 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:44.244280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:44.244379 (+ 99us) server_negotiation.cc:407] Connection header received
0924 10:35:44.244561 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:44.244564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:44.244634 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:44.244742 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:44.245364 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.246196 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.247547 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.247903 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.248038 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:44.251385 (+ 3347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.129:53277: BlockingRecv error: recv got EOF from 127.2.193.129:53277 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":60,"threads_started":1}
Sep 24 10:35:44 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:35:44 dist-test-slave-8wc7 krb5kdc[11001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.193.130@KRBTEST.COM for oryx/127.2.193.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:35:44.434706 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:44.428812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46035 (local address 127.2.193.130:47647)
0924 10:35:44.429162 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:44.429209 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:44.429354 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:44.429644 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:44.429647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:44.429874 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:44.429964 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:44.429970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.430787 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.430791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:44.431356 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:44.431365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.431868 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.431871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:44.431955 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:44.432541 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:44.432554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:44.434590 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.193.190:46035: Server oryx/127.2.193.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
W20260924 10:35:44.434901 12059 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:44.428988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:47647 (local address 127.2.193.190:46035)
0924 10:35:44.429157 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:44.429162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:44.429306 (+ 144us) server_negotiation.cc:407] Connection header received
0924 10:35:44.429468 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:44.429471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:44.429525 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:44.429619 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:44.430096 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.430663 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.431484 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.431714 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.431866 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:44.434764 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.193.130:47647: BlockingRecv error: recv got EOF from 127.2.193.130:47647 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:35:44.513551 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11584
I20260924 10:35:44.519093 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11734
I20260924 10:35:44.524327 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11868
I20260924 10:35:44.529372 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 11514
2026-09-24T10:35:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18052 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T10:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:44Z Disabled control of system clock
I20260924 10:35:44.559552 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:44891
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:35:44.670569 12064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:44.670853 12064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:44.670890 12064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:44.675031 12064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:44.675091 12064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:44.675107 12064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:44.675127 12064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:44.675141 12064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:44.679939 12064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:44891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:44.681082 12064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:44.682549 12064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:44.691438 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:44.691425 12072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:44.691432 12070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:44.691526 12064 server_base.cc:1034] running on GCE node
I20260924 10:35:44.692278 12064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:44.693512 12064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:44.694713 12064 hybrid_clock.cc:648] HybridClock initialized: now 1790246144694697 us; error 30 us; skew 500 ppm
I20260924 10:35:44.696688 12064 webserver.cc:460] Webserver started at http://127.2.193.190:34947/ using document root <none> and password file <none>
I20260924 10:35:44.697405 12064 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:44.697508 12064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:44.697750 12064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:44.699894 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "14de8a754d084653a99588c20c76edcf"
format_stamp: "Formatted at 2026-09-24 10:35:44 on dist-test-slave-8wc7"
server_key: "00fa53d15509bda7e9743d8a5f07bea6"
server_key_iv: "bc25546bdc296d514d59b2426ab134f4"
server_key_version: "encryptionkey@0"
I20260924 10:35:44.700489 12064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "14de8a754d084653a99588c20c76edcf"
format_stamp: "Formatted at 2026-09-24 10:35:44 on dist-test-slave-8wc7"
server_key: "00fa53d15509bda7e9743d8a5f07bea6"
server_key_iv: "bc25546bdc296d514d59b2426ab134f4"
server_key_version: "encryptionkey@0"
I20260924 10:35:44.704716 12064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260924 10:35:44.707319 12078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:44.708614 12064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:44.708765 12064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "14de8a754d084653a99588c20c76edcf"
format_stamp: "Formatted at 2026-09-24 10:35:44 on dist-test-slave-8wc7"
server_key: "00fa53d15509bda7e9743d8a5f07bea6"
server_key_iv: "bc25546bdc296d514d59b2426ab134f4"
server_key_version: "encryptionkey@0"
I20260924 10:35:44.708899 12064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:44.736835 12064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:44.737648 12064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:44.737877 12064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:44.745285 12064 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:44891
I20260924 10:35:44.745299 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:44891 every 8 connection(s)
I20260924 10:35:44.746338 12064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:44.749579 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:44.755039 12131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf: Bootstrap starting.
I20260924 10:35:44.755936 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12064
I20260924 10:35:44.756093 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:44.756438 2822 external_mini_cluster.cc:1428] Setting key 2ad079fb7f23978dc35e17a0752d948c
I20260924 10:35:44.758381 12131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:44.759158 12131 log.cc:824] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:44.761049 12131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf: No bootstrap required, opened a new log
I20260924 10:35:44.764451 12131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de8a754d084653a99588c20c76edcf" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 44891 } }
I20260924 10:35:44.764714 12131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:44.764782 12131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14de8a754d084653a99588c20c76edcf, State: Initialized, Role: FOLLOWER
I20260924 10:35:44.765344 12131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de8a754d084653a99588c20c76edcf" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 44891 } }
I20260924 10:35:44.765470 12131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:44.765558 12131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:44.765648 12131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:44.765761 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:44.757560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.2.193.190:44891)
0924 10:35:44.758181 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:44.758195 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:44.758231 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:35:44.758981 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:44.759000 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:44.759470 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:44.759834 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:44.760264 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.761311 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.762222 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.762712 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.762901 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:44.763791 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:44.763796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:44.763809 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:44.764203 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:44.764259 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:44.764281 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:44.764392 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:44.764509 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 10:35:44.764825 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
I20260924 10:35:44.766594 12131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de8a754d084653a99588c20c76edcf" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 44891 } }
I20260924 10:35:44.766955 12131 leader_election.cc:302] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14de8a754d084653a99588c20c76edcf; no voters:
I20260924 10:35:44.767313 12131 leader_election.cc:288] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:44.767429 12136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:44.767681 12136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [term 1 LEADER]: Becoming Leader. State: Replica: 14de8a754d084653a99588c20c76edcf, State: Running, Role: LEADER
I20260924 10:35:44.767997 12136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de8a754d084653a99588c20c76edcf" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 44891 } }
I20260924 10:35:44.768522 12131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:44.769443 12137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14de8a754d084653a99588c20c76edcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de8a754d084653a99588c20c76edcf" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 44891 } } }
I20260924 10:35:44.769585 12137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:44.769941 12145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:44.769933 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14de8a754d084653a99588c20c76edcf. Latest consensus state: current_term: 1 leader_uuid: "14de8a754d084653a99588c20c76edcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14de8a754d084653a99588c20c76edcf" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 44891 } } }
I20260924 10:35:44.770152 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:44.773666 12145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:44.779101 12145 catalog_manager.cc:1347] Generated new cluster ID: 881685a22b194dae8922e90076e8243f
I20260924 10:35:44.779189 12145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:44.793527 12145 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:44.794464 12145 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:44.805799 12145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14de8a754d084653a99588c20c76edcf: Generated new TSK 0
I20260924 10:35:44.806676 12145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:35:44.816620 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:44891
--builtin_ntp_servers=127.2.193.148:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 10:35:44.927064 12155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:44.927357 12155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:44.927395 12155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:44.931381 12155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:44.931438 12155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:44.931492 12155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:44.936218 12155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:44.937286 12155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:44.938668 12155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:44.948647 12163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:44.948657 12161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:44.948694 12160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:44.948879 12155 server_base.cc:1034] running on GCE node
I20260924 10:35:44.949431 12155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:44.950073 12155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:44.951229 12155 hybrid_clock.cc:648] HybridClock initialized: now 1790246144951210 us; error 32 us; skew 500 ppm
I20260924 10:35:44.953244 12155 webserver.cc:460] Webserver started at http://127.2.193.129:34451/ using document root <none> and password file <none>
I20260924 10:35:44.953838 12155 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:44.953902 12155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:44.954134 12155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:44.956151 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "500c3e7633624c6cb11f5d74fac32451"
format_stamp: "Formatted at 2026-09-24 10:35:44 on dist-test-slave-8wc7"
server_key: "500981b0b7cf1afbc052b2f73a7960ca"
server_key_iv: "056670fb9ff23fd6e1af57d17797b906"
server_key_version: "encryptionkey@0"
I20260924 10:35:44.956732 12155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "500c3e7633624c6cb11f5d74fac32451"
format_stamp: "Formatted at 2026-09-24 10:35:44 on dist-test-slave-8wc7"
server_key: "500981b0b7cf1afbc052b2f73a7960ca"
server_key_iv: "056670fb9ff23fd6e1af57d17797b906"
server_key_version: "encryptionkey@0"
I20260924 10:35:44.960420 12155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:35:44.962963 12169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:44.964162 12155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:35:44.964308 12155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "500c3e7633624c6cb11f5d74fac32451"
format_stamp: "Formatted at 2026-09-24 10:35:44 on dist-test-slave-8wc7"
server_key: "500981b0b7cf1afbc052b2f73a7960ca"
server_key_iv: "056670fb9ff23fd6e1af57d17797b906"
server_key_version: "encryptionkey@0"
I20260924 10:35:44.964439 12155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:44.976950 12155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:44.977757 12155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:44.977962 12155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:44.978541 12155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:44.979499 12155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:44.979569 12155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:44.979640 12155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:44.979692 12155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:44.988435 12155 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:33925
I20260924 10:35:44.988463 12282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:33925 every 8 connection(s)
I20260924 10:35:44.989410 12155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:44.992808 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12155
I20260924 10:35:44.992944 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:44.993283 2822 external_mini_cluster.cc:1428] Setting key 7a23ab9a9de530d1ea7898dd10534ae0
I20260924 10:35:44.995217 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:44891
--builtin_ntp_servers=127.2.193.148:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:35:44.999883 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:44.991323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:37103 (local address 127.2.193.190:44891)
0924 10:35:44.991473 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:44.991482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:44.992608 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 10:35:44.994453 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:44.994456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:44.994513 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:44.994645 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:44.995729 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.996329 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.997690 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.997942 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.998070 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:44.999323 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:44.999328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:44.999331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:44.999353 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:44.999373 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:44.999382 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:44.999470 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:44.999654 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:35:44.999723 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:35:45.000857 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:44.991723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:44891 (local address 127.2.193.129:37103)
0924 10:35:44.992436 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:44.992481 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:44.994215 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:44.994856 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:44.994864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:44.994961 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:44.995573 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:44.995589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.996476 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.996480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:44.997555 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:44.997565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:44.998113 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:44.998118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:44.998312 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:44.999022 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:44.999024 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:44.999071 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:44.999144 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:44.999472 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:44.999484 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:44.999493 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:35:44.999643 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 10:35:44.999981 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":114,"threads_started":1}
I20260924 10:35:45.002276 12283 heartbeater.cc:347] Connected to a master server at 127.2.193.190:44891
I20260924 10:35:45.002569 12283 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:45.003271 12283 heartbeater.cc:511] Master 127.2.193.190:44891 requested a full tablet report, sending...
I20260924 10:35:45.005123 12091 ts_manager.cc:194] Registered new tserver with Master: 500c3e7633624c6cb11f5d74fac32451 (127.2.193.129:33925)
I20260924 10:35:45.006791 12091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.129:37103
W20260924 10:35:45.108425 12286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:45.108709 12286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:45.108747 12286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:45.112855 12286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:45.112916 12286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:45.112974 12286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:45.117977 12286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:45.119057 12286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:45.120435 12286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:45.130162 12286 server_base.cc:1034] running on GCE node
W20260924 10:35:45.130108 12292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:45.130108 12291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:45.130125 12294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:45.130774 12286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:45.131472 12286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:45.132649 12286 hybrid_clock.cc:648] HybridClock initialized: now 1790246145132638 us; error 31 us; skew 500 ppm
I20260924 10:35:45.134617 12286 webserver.cc:460] Webserver started at http://127.2.193.130:34505/ using document root <none> and password file <none>
I20260924 10:35:45.135217 12286 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:45.135309 12286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:45.135558 12286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:45.137821 12286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "4b42322a6309473db8a23043831f76b6"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "c0c02d151a49830f916a4569057c688d"
server_key_iv: "eabdc7f77b9688264e7ac06c5434bbeb"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.138434 12286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "4b42322a6309473db8a23043831f76b6"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "c0c02d151a49830f916a4569057c688d"
server_key_iv: "eabdc7f77b9688264e7ac06c5434bbeb"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.142221 12286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:35:45.144945 12300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.146245 12286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:45.146397 12286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "4b42322a6309473db8a23043831f76b6"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "c0c02d151a49830f916a4569057c688d"
server_key_iv: "eabdc7f77b9688264e7ac06c5434bbeb"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.146522 12286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:45.180410 12286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:45.181241 12286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:45.181468 12286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:45.182107 12286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:45.183082 12286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:45.183157 12286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.183218 12286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:45.183271 12286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.192394 12286 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:40633
I20260924 10:35:45.192410 12413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:40633 every 8 connection(s)
I20260924 10:35:45.193375 12286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:45.201975 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12286
I20260924 10:35:45.202138 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:45.202546 2822 external_mini_cluster.cc:1428] Setting key eaea073f3063a925bb406f432f5642a7
I20260924 10:35:45.203228 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:45.195434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:54847 (local address 127.2.193.190:44891)
0924 10:35:45.195608 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:45.195611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:45.196730 (+ 1119us) server_negotiation.cc:407] Connection header received
0924 10:35:45.198247 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:45.198251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:45.198311 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:45.198431 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:45.199345 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.199885 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.201044 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.201293 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.201409 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:45.202651 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:45.202655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:45.202657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:45.202678 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:45.202697 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:45.202706 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:45.202769 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:45.202991 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 10:35:45.203076 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:35:45.204386 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:45.195859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:44891 (local address 127.2.193.130:54847)
0924 10:35:45.196555 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:45.196598 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:45.198020 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:45.198612 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:45.198620 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:45.198706 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:45.199198 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:45.199210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.200024 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.200032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:45.200902 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:45.200910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.201450 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.201454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:45.201624 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:45.202285 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:45.202287 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:45.202352 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:45.202451 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:45.202819 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:45.202828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:45.202836 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:35:45.202989 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 10:35:45.203323 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
I20260924 10:35:45.204838 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:44891
--builtin_ntp_servers=127.2.193.148:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:35:45.205585 12414 heartbeater.cc:347] Connected to a master server at 127.2.193.190:44891
I20260924 10:35:45.205854 12414 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:45.206504 12414 heartbeater.cc:511] Master 127.2.193.190:44891 requested a full tablet report, sending...
I20260924 10:35:45.207765 12091 ts_manager.cc:194] Registered new tserver with Master: 4b42322a6309473db8a23043831f76b6 (127.2.193.130:40633)
I20260924 10:35:45.208629 12091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.130:54847
W20260924 10:35:45.322288 12417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:45.322563 12417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:45.322602 12417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:45.326951 12417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:45.327009 12417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:45.327065 12417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:45.331936 12417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:45.333001 12417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:45.334458 12417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:35:45.344625 12417 server_base.cc:1034] running on GCE node
W20260924 10:35:45.344626 12422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:45.344787 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:45.344671 12423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:45.345412 12417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:45.346030 12417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:45.347208 12417 hybrid_clock.cc:648] HybridClock initialized: now 1790246145347183 us; error 38 us; skew 500 ppm
I20260924 10:35:45.349239 12417 webserver.cc:460] Webserver started at http://127.2.193.131:37747/ using document root <none> and password file <none>
I20260924 10:35:45.349889 12417 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:45.349954 12417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:45.350193 12417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:45.352423 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "82b23d1a4a264c4ab7e4164151c7fef7"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "9bbbc2d9541c107af970740875727465"
server_key_iv: "94b284f086e1f967ac53958e5d01be1c"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.353006 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "82b23d1a4a264c4ab7e4164151c7fef7"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "9bbbc2d9541c107af970740875727465"
server_key_iv: "94b284f086e1f967ac53958e5d01be1c"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.356721 12417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:35:45.359448 12431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.360795 12417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:45.360939 12417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "82b23d1a4a264c4ab7e4164151c7fef7"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "9bbbc2d9541c107af970740875727465"
server_key_iv: "94b284f086e1f967ac53958e5d01be1c"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.361061 12417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:45.376663 12417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:45.377542 12417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:45.377767 12417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:45.378420 12417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:45.379429 12417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:45.379477 12417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.379549 12417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:45.379585 12417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.388396 12417 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:39293
I20260924 10:35:45.388417 12544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:39293 every 8 connection(s)
I20260924 10:35:45.389407 12417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:45.391932 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12417
I20260924 10:35:45.392055 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:45.392391 2822 external_mini_cluster.cc:1428] Setting key b191e8f37e363a50d35a5e225f585e4f
I20260924 10:35:45.399222 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:45.391366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:59209 (local address 127.2.193.190:44891)
0924 10:35:45.391498 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:45.391503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:45.392827 (+ 1324us) server_negotiation.cc:407] Connection header received
0924 10:35:45.394497 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:45.394500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:45.394556 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:45.394634 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:45.395586 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.396106 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.397337 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.397576 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.397661 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:45.398816 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:45.398819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:45.398821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:45.398832 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:45.398848 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:45.398855 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:45.398909 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:45.399047 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 10:35:45.399102 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:35:45.400192 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:45.391764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:44891 (local address 127.2.193.131:59209)
0924 10:35:45.392627 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:45.392691 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:45.394231 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:45.394813 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:45.394820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:45.394913 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:45.395458 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:45.395471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.396221 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.396224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:45.397157 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:45.397226 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:45.397723 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:45.397726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:45.397874 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:45.398559 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:45.398560 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:45.398608 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:45.398680 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:45.398924 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:45.398930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:45.398934 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:35:45.399035 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:35:45.399316 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":155,"threads_started":1}
I20260924 10:35:45.401407 12545 heartbeater.cc:347] Connected to a master server at 127.2.193.190:44891
I20260924 10:35:45.401665 12545 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:45.402233 12545 heartbeater.cc:511] Master 127.2.193.190:44891 requested a full tablet report, sending...
I20260924 10:35:45.403260 12091 ts_manager.cc:194] Registered new tserver with Master: 82b23d1a4a264c4ab7e4164151c7fef7 (127.2.193.131:39293)
I20260924 10:35:45.403836 12091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.131:59209
I20260924 10:35:45.406323 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:35:45.409348 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:45.408322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.2.193.190:44891)
0924 10:35:45.408495 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:45.408500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:45.408646 (+ 146us) server_negotiation.cc:407] Connection header received
0924 10:35:45.408807 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:45.408812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:45.408858 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:45.408947 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:45.409233 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54806: BlockingRecv error: recv got EOF from 127.0.0.1:54806 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 10:35:45.409646 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12155
I20260924 10:35:45.415253 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12286
I20260924 10:35:45.420504 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12417
I20260924 10:35:45.425762 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12064
2026-09-24T10:35:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (895 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T10:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:45Z Disabled control of system clock
I20260924 10:35:45.455715 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46823
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:35:45.570705 12556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:45.571089 12556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:45.571136 12556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:45.575381 12556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:45.575459 12556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:45.575474 12556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:45.575492 12556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:45.575507 12556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:45.580334 12556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:45.581561 12556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:45.582978 12556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:45.591583 12561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:45.591581 12562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:45.591759 12556 server_base.cc:1034] running on GCE node
W20260924 10:35:45.591773 12564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:45.592535 12556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:45.593719 12556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:45.594884 12556 hybrid_clock.cc:648] HybridClock initialized: now 1790246145594873 us; error 29 us; skew 500 ppm
I20260924 10:35:45.596777 12556 webserver.cc:460] Webserver started at http://127.2.193.190:45325/ using document root <none> and password file <none>
I20260924 10:35:45.597476 12556 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:45.597546 12556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:45.597769 12556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:45.599898 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "374ed64b3edc4b30aea062b15c0b3ffc"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "e88d97bc053fd579f626428cd8615cc6"
server_key_iv: "679eeb9dcd2d64ec550df0ce18b15983"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.600489 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "374ed64b3edc4b30aea062b15c0b3ffc"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "e88d97bc053fd579f626428cd8615cc6"
server_key_iv: "679eeb9dcd2d64ec550df0ce18b15983"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.604316 12556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:35:45.606824 12570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.608116 12556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:35:45.608263 12556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "374ed64b3edc4b30aea062b15c0b3ffc"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "e88d97bc053fd579f626428cd8615cc6"
server_key_iv: "679eeb9dcd2d64ec550df0ce18b15983"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.608385 12556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:45.613063 12556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:45.613734 12556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:45.613929 12556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:45.620661 12556 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:46823
I20260924 10:35:45.620705 12622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:46823 every 8 connection(s)
I20260924 10:35:45.621717 12556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:45.621981 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12556
I20260924 10:35:45.622102 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:45.622445 2822 external_mini_cluster.cc:1428] Setting key c2a7bd962f15ff53dc0c68a6f24b76ec
I20260924 10:35:45.625389 12623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:45.629056 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:45.624005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.2.193.190:46823)
0924 10:35:45.625338 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:45.625361 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:45.625400 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:35:45.626128 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:45.626147 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:45.626527 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:45.626916 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:45.627753 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:45.627757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:45.627769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:45.627779 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:45.627836 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:45.627852 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:45.628021 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:45.628093 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 10:35:45.628341 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":712,"threads_started":1}
I20260924 10:35:45.632007 12623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc: Bootstrap starting.
I20260924 10:35:45.634434 12623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:45.635390 12623 log.cc:824] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:45.637506 12623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc: No bootstrap required, opened a new log
I20260924 10:35:45.640980 12623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46823 } }
I20260924 10:35:45.641302 12623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:45.641383 12623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374ed64b3edc4b30aea062b15c0b3ffc, State: Initialized, Role: FOLLOWER
I20260924 10:35:45.641885 12623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46823 } }
I20260924 10:35:45.642019 12623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:45.642067 12623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:45.642139 12623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:45.643055 12623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46823 } }
I20260924 10:35:45.643445 12623 leader_election.cc:302] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 374ed64b3edc4b30aea062b15c0b3ffc; no voters:
I20260924 10:35:45.643769 12623 leader_election.cc:288] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:45.643925 12628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:45.644169 12628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [term 1 LEADER]: Becoming Leader. State: Replica: 374ed64b3edc4b30aea062b15c0b3ffc, State: Running, Role: LEADER
I20260924 10:35:45.644476 12628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46823 } }
I20260924 10:35:45.645139 12623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:45.646198 12629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46823 } } }
I20260924 10:35:45.646371 12629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:45.646220 12630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 374ed64b3edc4b30aea062b15c0b3ffc. Latest consensus state: current_term: 1 leader_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374ed64b3edc4b30aea062b15c0b3ffc" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46823 } } }
I20260924 10:35:45.646620 12630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:45.646839 12637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:45.649693 12637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:45.655531 12637 catalog_manager.cc:1347] Generated new cluster ID: 7569714298c7491d8bd62e39a3f510dc
I20260924 10:35:45.655620 12637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:45.667534 12637 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:45.667902 12637 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:45.673393 12637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 374ed64b3edc4b30aea062b15c0b3ffc: Generated new TSK 0
I20260924 10:35:45.674299 12637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:35:45.685467 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46823
--builtin_ntp_servers=127.2.193.148:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:35:45.805074 12647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:45.805387 12647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:45.805428 12647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:45.809502 12647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:45.809571 12647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:45.809631 12647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:45.814602 12647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:45.815678 12647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:45.817318 12647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:45.827085 12652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:45.827185 12653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:45.827190 12647 server_base.cc:1034] running on GCE node
W20260924 10:35:45.827095 12655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:45.827886 12647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:45.828538 12647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:45.829731 12647 hybrid_clock.cc:648] HybridClock initialized: now 1790246145829717 us; error 28 us; skew 500 ppm
I20260924 10:35:45.831575 12647 webserver.cc:460] Webserver started at http://127.2.193.129:32871/ using document root <none> and password file <none>
I20260924 10:35:45.832181 12647 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:45.832275 12647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:45.832494 12647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:45.834626 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "9168999793894272b529c34a6a015db2"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "606c24f32ad5af6c81231091d69da129"
server_key_iv: "b3dfcde94068bc4d4536883534001fbd"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.835211 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "9168999793894272b529c34a6a015db2"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "606c24f32ad5af6c81231091d69da129"
server_key_iv: "b3dfcde94068bc4d4536883534001fbd"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.839011 12647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:45.841670 12661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.842882 12647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:45.843029 12647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "9168999793894272b529c34a6a015db2"
format_stamp: "Formatted at 2026-09-24 10:35:45 on dist-test-slave-8wc7"
server_key: "606c24f32ad5af6c81231091d69da129"
server_key_iv: "b3dfcde94068bc4d4536883534001fbd"
server_key_version: "encryptionkey@0"
I20260924 10:35:45.843151 12647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:45.848008 12647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:45.848627 12647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:45.848807 12647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:45.849423 12647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:45.850322 12647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:45.850394 12647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.850456 12647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:45.850509 12647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:45.858875 12647 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:33059
I20260924 10:35:45.858891 12774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:33059 every 8 connection(s)
I20260924 10:35:45.859804 12647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:45.862046 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12647
I20260924 10:35:45.862164 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:45.862511 2822 external_mini_cluster.cc:1428] Setting key 4a460ed900ff8546ab093abbfcb78b03
I20260924 10:35:45.864468 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46823
--builtin_ntp_servers=127.2.193.148:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:35:45.866427 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:45.861723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:47943 (local address 127.2.193.190:46823)
0924 10:35:45.861910 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:45.861915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:45.863086 (+ 1171us) server_negotiation.cc:407] Connection header received
0924 10:35:45.864544 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:45.864547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:45.864588 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:45.864663 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:45.865963 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:45.865968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:45.865971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:45.865991 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:45.866011 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:45.866019 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:45.866097 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:45.866267 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 10:35:45.866303 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:35:45.867585 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:45.862203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46823 (local address 127.2.193.129:47943)
0924 10:35:45.862933 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:45.862984 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:45.864336 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:45.864851 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:45.864858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:45.864949 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:45.865574 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:45.865576 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:45.865616 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:45.865726 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:45.866115 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:45.866124 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:45.866133 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:35:45.866309 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 10:35:45.866565 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260924 10:35:45.868862 12775 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46823
I20260924 10:35:45.869112 12775 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:45.869551 12775 heartbeater.cc:511] Master 127.2.193.190:46823 requested a full tablet report, sending...
I20260924 10:35:45.871038 12587 ts_manager.cc:194] Registered new tserver with Master: 9168999793894272b529c34a6a015db2 (127.2.193.129:33059)
W20260924 10:35:45.978293 12778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:45.978559 12778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:45.978597 12778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:45.982595 12778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:45.982647 12778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:45.982741 12778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:45.987541 12778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:45.988678 12778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:45.990079 12778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:45.999537 12786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:45.999555 12783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:45.999776 12778 server_base.cc:1034] running on GCE node
W20260924 10:35:45.999594 12784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:46.000224 12778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:46.000878 12778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:46.002072 12778 hybrid_clock.cc:648] HybridClock initialized: now 1790246146002056 us; error 25 us; skew 500 ppm
I20260924 10:35:46.003984 12778 webserver.cc:460] Webserver started at http://127.2.193.130:33903/ using document root <none> and password file <none>
I20260924 10:35:46.004573 12778 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:46.004674 12778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:46.004910 12778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:46.006970 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "0196a3eec71844368a9e22e90d39de84"
format_stamp: "Formatted at 2026-09-24 10:35:46 on dist-test-slave-8wc7"
server_key: "1cb8f2a9ae67b9aa5ef78ec9cfc99e2e"
server_key_iv: "90a33d285ad1c74575f2e9bb134cbb7a"
server_key_version: "encryptionkey@0"
I20260924 10:35:46.007535 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "0196a3eec71844368a9e22e90d39de84"
format_stamp: "Formatted at 2026-09-24 10:35:46 on dist-test-slave-8wc7"
server_key: "1cb8f2a9ae67b9aa5ef78ec9cfc99e2e"
server_key_iv: "90a33d285ad1c74575f2e9bb134cbb7a"
server_key_version: "encryptionkey@0"
I20260924 10:35:46.011145 12778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:35:46.013712 12792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:46.014896 12778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:46.015023 12778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "0196a3eec71844368a9e22e90d39de84"
format_stamp: "Formatted at 2026-09-24 10:35:46 on dist-test-slave-8wc7"
server_key: "1cb8f2a9ae67b9aa5ef78ec9cfc99e2e"
server_key_iv: "90a33d285ad1c74575f2e9bb134cbb7a"
server_key_version: "encryptionkey@0"
I20260924 10:35:46.015136 12778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:46.019701 12778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:46.020283 12778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:46.020457 12778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:46.020969 12778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:46.021890 12778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:46.021957 12778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:46.022020 12778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:46.022063 12778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:46.030701 12778 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:38039
I20260924 10:35:46.030720 12905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:38039 every 8 connection(s)
I20260924 10:35:46.031580 12778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:46.037129 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:46.033394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:56591 (local address 127.2.193.190:46823)
0924 10:35:46.033542 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:46.033546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:46.034480 (+ 934us) server_negotiation.cc:407] Connection header received
0924 10:35:46.035698 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:46.035700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:46.035735 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:46.035813 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:46.036760 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:46.036763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:46.036765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:46.036775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:46.036792 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:46.036811 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:46.036884 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:46.036995 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 10:35:46.037029 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:35:46.037947 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:46.033750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46823 (local address 127.2.193.130:56591)
0924 10:35:46.034362 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:46.034399 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:46.035521 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:46.035958 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:46.035964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:46.036044 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:46.036534 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:46.036536 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:46.036572 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:46.036632 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:46.036884 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:46.036889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:46.036893 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:35:46.037010 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:35:46.037233 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260924 10:35:46.039022 12906 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46823
I20260924 10:35:46.039230 12906 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:46.039604 12906 heartbeater.cc:511] Master 127.2.193.190:46823 requested a full tablet report, sending...
I20260924 10:35:46.040604 12587 ts_manager.cc:194] Registered new tserver with Master: 0196a3eec71844368a9e22e90d39de84 (127.2.193.130:38039)
I20260924 10:35:46.041242 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12778
I20260924 10:35:46.041363 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:46.041657 2822 external_mini_cluster.cc:1428] Setting key 3692d883844d938074dda4e3e5e3b404
I20260924 10:35:46.043380 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46823
--builtin_ntp_servers=127.2.193.148:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:35:46.155107 12909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:46.155390 12909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:46.155431 12909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:46.159540 12909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:46.159598 12909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:46.159654 12909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:46.164460 12909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:46.165633 12909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:46.167043 12909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:46.177062 12914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:46.177058 12917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:46.177088 12915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:46.177556 12909 server_base.cc:1034] running on GCE node
I20260924 10:35:46.177989 12909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:46.178637 12909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:46.179832 12909 hybrid_clock.cc:648] HybridClock initialized: now 1790246146179812 us; error 32 us; skew 500 ppm
I20260924 10:35:46.181867 12909 webserver.cc:460] Webserver started at http://127.2.193.131:43153/ using document root <none> and password file <none>
I20260924 10:35:46.182626 12909 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:46.182729 12909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:46.182979 12909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:46.185091 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "08d6a7bdaecc4692a94aa94563ed7c75"
format_stamp: "Formatted at 2026-09-24 10:35:46 on dist-test-slave-8wc7"
server_key: "c04cedb9ae3551250893ff920a5e454a"
server_key_iv: "35a4d71fccb9fb83ffed602c08453f30"
server_key_version: "encryptionkey@0"
I20260924 10:35:46.185729 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "08d6a7bdaecc4692a94aa94563ed7c75"
format_stamp: "Formatted at 2026-09-24 10:35:46 on dist-test-slave-8wc7"
server_key: "c04cedb9ae3551250893ff920a5e454a"
server_key_iv: "35a4d71fccb9fb83ffed602c08453f30"
server_key_version: "encryptionkey@0"
I20260924 10:35:46.189538 12909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:35:46.192220 12923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:46.193673 12909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:46.193826 12909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "08d6a7bdaecc4692a94aa94563ed7c75"
format_stamp: "Formatted at 2026-09-24 10:35:46 on dist-test-slave-8wc7"
server_key: "c04cedb9ae3551250893ff920a5e454a"
server_key_iv: "35a4d71fccb9fb83ffed602c08453f30"
server_key_version: "encryptionkey@0"
I20260924 10:35:46.193953 12909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:46.198655 12909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:46.199281 12909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:46.199468 12909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:46.200035 12909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:46.200963 12909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:46.201037 12909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:46.201103 12909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:46.201150 12909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:46.210474 12909 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:35847
I20260924 10:35:46.210494 13036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:35847 every 8 connection(s)
I20260924 10:35:46.211550 12909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:46.218079 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:46.213548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:52461 (local address 127.2.193.190:46823)
0924 10:35:46.213685 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:46.213689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:46.214770 (+ 1081us) server_negotiation.cc:407] Connection header received
0924 10:35:46.216112 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:46.216115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:46.216163 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:46.216242 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:46.217543 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:46.217547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:46.217550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:35:46.217563 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:46.217587 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:35:46.217597 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:46.217705 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:46.217859 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 10:35:46.217909 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:35:46.218927 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:46.213982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46823 (local address 127.2.193.131:52461)
0924 10:35:46.214616 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:46.214657 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:46.215882 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:46.216409 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:46.216416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:46.216507 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:46.217136 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:35:46.217138 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:46.217249 (+ 111us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:35:46.217363 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:46.217684 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:46.217692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:46.217698 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:35:46.217900 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 10:35:46.218148 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260924 10:35:46.219928 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 12909
I20260924 10:35:46.220038 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:46.220176 13037 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46823
I20260924 10:35:46.220397 2822 external_mini_cluster.cc:1428] Setting key ea66c793841f7b0f22b9d5b820746f60
I20260924 10:35:46.220448 13037 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:46.220937 13037 heartbeater.cc:511] Master 127.2.193.190:46823 requested a full tablet report, sending...
I20260924 10:35:46.222103 12586 ts_manager.cc:194] Registered new tserver with Master: 08d6a7bdaecc4692a94aa94563ed7c75 (127.2.193.131:35847)
I20260924 10:35:46.234822 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:35:46.237900 12626 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:35:46.236809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.2.193.190:46823)
0924 10:35:46.236980 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:46.236987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:46.237161 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:35:46.237345 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:46.237352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:46.237395 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:46.237487 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:46.237743 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54480: BlockingRecv error: recv got EOF from 127.0.0.1:54480 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:35:46.238163 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12647
I20260924 10:35:46.244701 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12778
I20260924 10:35:46.251817 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12909
I20260924 10:35:46.256848 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 12556
2026-09-24T10:35:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:46 dist-test-slave-8wc7 krb5kdc[13047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:46 dist-test-slave-8wc7 krb5kdc[13047](info): set up 2 sockets
Sep 24 10:35:46 dist-test-slave-8wc7 krb5kdc[13047](info): commencing operation
krb5kdc: starting...
W20260924 10:35:47.290278 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:47 dist-test-slave-8wc7 krb5kdc[13047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:35:47.421718 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40447
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:47.534215 13060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:47.534556 13060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:47.534644 13060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:47.538818 13060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:35:47.538908 13060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:47.538946 13060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:47.539006 13060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:35:47.539047 13060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:35:47.544421 13060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:47.545626 13060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:47.546825 13060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:47.555714 13065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:47.555694 13060 server_base.cc:1034] running on GCE node
W20260924 10:35:47.555683 13066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:47.555678 13068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:47.556519 13060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:47.557749 13060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:47.558964 13060 hybrid_clock.cc:648] HybridClock initialized: now 1790246147558949 us; error 27 us; skew 500 ppm
Sep 24 10:35:47 dist-test-slave-8wc7 krb5kdc[13047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246147, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:47.561985 13060 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:35:47.562968 13060 webserver.cc:460] Webserver started at http://127.2.193.190:34921/ using document root <none> and password file <none>
I20260924 10:35:47.563552 13060 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:47.563647 13060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:47.563895 13060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:47.566459 13060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "b7b22724ce064e64894dddc2aa37aa3a"
format_stamp: "Formatted at 2026-09-24 10:35:47 on dist-test-slave-8wc7"
server_key: "7fd0b5e697a8ccc1103669de991900e2"
server_key_iv: "f234a05cad70a0de33e6f46a0bc142ee"
server_key_version: "encryptionkey@0"
I20260924 10:35:47.567080 13060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "b7b22724ce064e64894dddc2aa37aa3a"
format_stamp: "Formatted at 2026-09-24 10:35:47 on dist-test-slave-8wc7"
server_key: "7fd0b5e697a8ccc1103669de991900e2"
server_key_iv: "f234a05cad70a0de33e6f46a0bc142ee"
server_key_version: "encryptionkey@0"
I20260924 10:35:47.571244 13060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:35:47.573868 13075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:47.575248 13060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:47.575449 13060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "b7b22724ce064e64894dddc2aa37aa3a"
format_stamp: "Formatted at 2026-09-24 10:35:47 on dist-test-slave-8wc7"
server_key: "7fd0b5e697a8ccc1103669de991900e2"
server_key_iv: "f234a05cad70a0de33e6f46a0bc142ee"
server_key_version: "encryptionkey@0"
I20260924 10:35:47.575595 13060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:47.594043 13060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:47.597270 13060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:47.597435 13060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:47.604517 13060 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:40447
I20260924 10:35:47.604805 13127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:40447 every 8 connection(s)
I20260924 10:35:47.605695 13060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:35:47.608076 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 13060
I20260924 10:35:47.608170 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:35:47.608440 2822 external_mini_cluster.cc:1428] Setting key 55fa9fccbd82e6eb3a1c43f4b3332ac8
I20260924 10:35:47.608600 13128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:47.615005 13128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a: Bootstrap starting.
Sep 24 10:35:47 dist-test-slave-8wc7 krb5kdc[13047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:47.617463 13128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:47.618211 13128 log.cc:824] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:47.620239 13128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a: No bootstrap required, opened a new log
I20260924 10:35:47.622720 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:47.609914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.2.193.190:40447)
0924 10:35:47.610341 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:47.610356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:47.610392 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:35:47.611284 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:47.611303 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:47.611636 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:47.611976 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:47.612776 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:47.613927 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:47.614765 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:47.615314 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:47.615503 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:47.617995 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:47.618000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:47.618013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:47.618040 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:47.619729 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:47.620180 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:47.620185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:47.620190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:47.620459 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:47.620781 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:47.620786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:47.620788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:47.621225 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:47.621424 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:47.621715 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 10:35:47.622002 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":141,"threads_started":1}
I20260924 10:35:47.624059 13128 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b22724ce064e64894dddc2aa37aa3a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40447 } }
I20260924 10:35:47.624275 13128 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:47.624346 13128 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b22724ce064e64894dddc2aa37aa3a, State: Initialized, Role: FOLLOWER
I20260924 10:35:47.624754 13128 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b22724ce064e64894dddc2aa37aa3a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40447 } }
I20260924 10:35:47.624941 13128 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:35:47.625006 13128 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:35:47.625103 13128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:47.626041 13128 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b22724ce064e64894dddc2aa37aa3a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40447 } }
I20260924 10:35:47.626417 13128 leader_election.cc:302] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b22724ce064e64894dddc2aa37aa3a; no voters:
I20260924 10:35:47.626780 13128 leader_election.cc:288] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:35:47.626866 13133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:47.627058 13133 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [term 1 LEADER]: Becoming Leader. State: Replica: b7b22724ce064e64894dddc2aa37aa3a, State: Running, Role: LEADER
I20260924 10:35:47.627378 13133 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b22724ce064e64894dddc2aa37aa3a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40447 } }
I20260924 10:35:47.628129 13128 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:35:47.629251 13135 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b22724ce064e64894dddc2aa37aa3a. Latest consensus state: current_term: 1 leader_uuid: "b7b22724ce064e64894dddc2aa37aa3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b22724ce064e64894dddc2aa37aa3a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40447 } } }
I20260924 10:35:47.629266 13134 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b22724ce064e64894dddc2aa37aa3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b22724ce064e64894dddc2aa37aa3a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40447 } } }
I20260924 10:35:47.629457 13134 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:47.629458 13135 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a [sys.catalog]: This master's current role is: LEADER
I20260924 10:35:47.630705 13142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:35:47.633481 13142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:35:47.638594 13142 catalog_manager.cc:1347] Generated new cluster ID: 88f6f43c921e4afb954df792d32ab471
I20260924 10:35:47.638662 13142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:35:47.662523 13142 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:35:47.663622 13142 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:35:47.668443 13142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a: Generated new TSK 0
I20260924 10:35:47.669230 13142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:35:47.704733 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40447
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:47.820571 13154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:47.821007 13154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:47.821069 13154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:47.825098 13154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:47.825165 13154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:47.825242 13154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:35:47.830803 13154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:47.831836 13154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:47.832816 13154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:47.842283 13159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:47.842303 13160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:47.842293 13162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:47.842391 13154 server_base.cc:1034] running on GCE node
I20260924 10:35:47.842903 13154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:47.843554 13154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:47.844743 13154 hybrid_clock.cc:648] HybridClock initialized: now 1790246147844708 us; error 47 us; skew 500 ppm
Sep 24 10:35:47 dist-test-slave-8wc7 krb5kdc[13047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246147, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:47.847579 13154 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:35:47.848451 13154 webserver.cc:460] Webserver started at http://127.2.193.129:43455/ using document root <none> and password file <none>
I20260924 10:35:47.849130 13154 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:47.849241 13154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:47.849479 13154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:47.851547 13154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "d9f7b006a0a948b792c2ddebb745f50b"
format_stamp: "Formatted at 2026-09-24 10:35:47 on dist-test-slave-8wc7"
server_key: "ee5657b7cbe51a4ae5c7e892233bc2d3"
server_key_iv: "5f0f831d0d47b97b125a74f6763a4df0"
server_key_version: "encryptionkey@0"
I20260924 10:35:47.852063 13154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "d9f7b006a0a948b792c2ddebb745f50b"
format_stamp: "Formatted at 2026-09-24 10:35:47 on dist-test-slave-8wc7"
server_key: "ee5657b7cbe51a4ae5c7e892233bc2d3"
server_key_iv: "5f0f831d0d47b97b125a74f6763a4df0"
server_key_version: "encryptionkey@0"
I20260924 10:35:47.855717 13154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:35:47.858124 13169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:47.859380 13154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:47.859501 13154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "d9f7b006a0a948b792c2ddebb745f50b"
format_stamp: "Formatted at 2026-09-24 10:35:47 on dist-test-slave-8wc7"
server_key: "ee5657b7cbe51a4ae5c7e892233bc2d3"
server_key_iv: "5f0f831d0d47b97b125a74f6763a4df0"
server_key_version: "encryptionkey@0"
I20260924 10:35:47.859608 13154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:47.872426 13154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:47.875831 13154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:47.876042 13154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:47.876644 13154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:47.877584 13154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:47.877629 13154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:47.877697 13154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:47.877729 13154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:47.886446 13154 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:45207
I20260924 10:35:47.886507 13282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:45207 every 8 connection(s)
I20260924 10:35:47.887377 13154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:35:47.891770 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 13154
I20260924 10:35:47.891886 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:35:47.892210 2822 external_mini_cluster.cc:1428] Setting key c47c7d9de1cf3060cfedc2b80911e8f9
Sep 24 10:35:47 dist-test-slave-8wc7 krb5kdc[13047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246147, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:35:47.902753 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:47.889243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:53413 (local address 127.2.193.190:40447)
0924 10:35:47.889439 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:47.889445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:47.890265 (+ 820us) server_negotiation.cc:407] Connection header received
0924 10:35:47.891577 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:47.891581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:47.891632 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:47.891724 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:47.893288 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:47.893992 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:47.895757 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:47.896163 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:47.896280 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:47.898857 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:47.898863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:47.898868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:47.898904 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:47.900546 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:47.901034 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:47.901038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:47.901041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:47.901138 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:47.901518 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:47.901523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:47.901528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:47.901779 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:47.901898 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:47.902390 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 10:35:47.902527 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 10:35:47.903575 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:47.889670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40447 (local address 127.2.193.129:53413)
0924 10:35:47.890110 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:47.890153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:47.891337 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:47.891937 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:47.891946 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:47.892451 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:47.893102 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:47.893117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:47.894127 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:47.894131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:47.895636 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:47.895646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:47.896310 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:47.896312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:47.896449 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:47.897084 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:47.897100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:47.898671 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:47.900658 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:47.900663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:47.900675 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:47.900919 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:47.901276 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:47.901278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:47.901279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:47.901407 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:47.901897 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:47.901902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:47.902138 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 10:35:47.902283 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:35:47.902609 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":127,"threads_started":1}
I20260924 10:35:47.904877 13283 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40447
I20260924 10:35:47.905129 13283 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:35:47.905723 13283 heartbeater.cc:511] Master 127.2.193.190:40447 requested a full tablet report, sending...
I20260924 10:35:47.907584 13092 ts_manager.cc:194] Registered new tserver with Master: d9f7b006a0a948b792c2ddebb745f50b (127.2.193.129:45207)
I20260924 10:35:47.908804 13092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:53413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:35:47.922688 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40447
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:48.036772 13288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:48.037107 13288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:48.037235 13288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:48.041510 13288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:48.041594 13288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:48.041687 13288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:35:48.047070 13288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:48.048270 13288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:48.049459 13288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:48.059178 13294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:48.059229 13293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:48.059337 13288 server_base.cc:1034] running on GCE node
W20260924 10:35:48.059178 13296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:48.059937 13288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:48.060591 13288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:48.061779 13288 hybrid_clock.cc:648] HybridClock initialized: now 1790246148061758 us; error 32 us; skew 500 ppm
Sep 24 10:35:48 dist-test-slave-8wc7 krb5kdc[13047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246148, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:48.064641 13288 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:35:48.065534 13288 webserver.cc:460] Webserver started at http://127.2.193.130:44799/ using document root <none> and password file <none>
I20260924 10:35:48.066090 13288 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:48.066177 13288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:48.066409 13288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:48.068506 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "0e30f24383a94f5091ac4c432d3ebc05"
format_stamp: "Formatted at 2026-09-24 10:35:48 on dist-test-slave-8wc7"
server_key: "6b202c6d968c27dda46ee5a5e5e65070"
server_key_iv: "9b4a26e881763ce4cfe0c3b02ba3353e"
server_key_version: "encryptionkey@0"
I20260924 10:35:48.069078 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "0e30f24383a94f5091ac4c432d3ebc05"
format_stamp: "Formatted at 2026-09-24 10:35:48 on dist-test-slave-8wc7"
server_key: "6b202c6d968c27dda46ee5a5e5e65070"
server_key_iv: "9b4a26e881763ce4cfe0c3b02ba3353e"
server_key_version: "encryptionkey@0"
I20260924 10:35:48.072861 13288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:35:48.075372 13303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:48.076658 13288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:35:48.076810 13288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "0e30f24383a94f5091ac4c432d3ebc05"
format_stamp: "Formatted at 2026-09-24 10:35:48 on dist-test-slave-8wc7"
server_key: "6b202c6d968c27dda46ee5a5e5e65070"
server_key_iv: "9b4a26e881763ce4cfe0c3b02ba3353e"
server_key_version: "encryptionkey@0"
I20260924 10:35:48.076941 13288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:48.095480 13288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:48.098532 13288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:48.098745 13288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:48.099390 13288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:48.100344 13288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:48.100415 13288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:48.100486 13288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:48.100540 13288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:48.110080 13288 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:46809
I20260924 10:35:48.110100 13416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:46809 every 8 connection(s)
I20260924 10:35:48.110999 13288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:35:48.119602 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 13288
I20260924 10:35:48.119699 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:35:48.119983 2822 external_mini_cluster.cc:1428] Setting key 410a0647bca60df78e44cf8fcfcc7a5a
Sep 24 10:35:48 dist-test-slave-8wc7 krb5kdc[13047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246148, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:48.125841 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.112769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:35269 (local address 127.2.193.190:40447)
0924 10:35:48.112899 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.112902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.113742 (+ 840us) server_negotiation.cc:407] Connection header received
0924 10:35:48.115123 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.115126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.115190 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.115298 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:48.116493 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.117027 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.118213 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.118448 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.118573 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.121506 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:48.121510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:48.121513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:48.121541 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:48.123335 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:48.124018 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:48.124021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:48.124022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:48.124104 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:48.124508 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:48.124510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:48.124511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:48.124763 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:48.124865 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.125448 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.125597 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:35:48.126523 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.113106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40447 (local address 127.2.193.130:35269)
0924 10:35:48.113580 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.113625 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:48.114896 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:48.115476 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:48.115483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:48.115843 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:48.116335 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.116348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.117156 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.117160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.118083 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.118093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.118613 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.118617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.118791 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.119465 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:48.119481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:48.121241 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:48.123490 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:48.123501 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:48.123521 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:48.123896 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:48.124197 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:48.124201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:48.124203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:48.124403 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:48.124865 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:48.124871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:48.125122 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 10:35:48.125345 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 10:35:48.125619 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260924 10:35:48.127799 13417 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40447
I20260924 10:35:48.128100 13417 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:48.128759 13417 heartbeater.cc:511] Master 127.2.193.190:40447 requested a full tablet report, sending...
I20260924 10:35:48.130054 13092 ts_manager.cc:194] Registered new tserver with Master: 0e30f24383a94f5091ac4c432d3ebc05 (127.2.193.130:46809)
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:35:48.130658 13092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:35269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:35:48.148028 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40447
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:35:48.263229 13422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:35:48.263494 13422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:35:48.263540 13422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:35:48.267733 13422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:35:48.267797 13422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:35:48.267858 13422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:35:48.273164 13422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:35:48.274214 13422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:35:48.275194 13422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:35:48.284814 13430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:35:48.284878 13428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:48.284991 13422 server_base.cc:1034] running on GCE node
W20260924 10:35:48.284873 13427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:35:48.285521 13422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:35:48.286144 13422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:35:48.287313 13422 hybrid_clock.cc:648] HybridClock initialized: now 1790246148287296 us; error 27 us; skew 500 ppm
Sep 24 10:35:48 dist-test-slave-8wc7 krb5kdc[13047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246148, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:35:48.299006 13422 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:35:48.300052 13422 webserver.cc:460] Webserver started at http://127.2.193.131:34131/ using document root <none> and password file <none>
I20260924 10:35:48.300679 13422 fs_manager.cc:359] Metadata directory not provided
I20260924 10:35:48.300774 13422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:35:48.301018 13422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:35:48.303222 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "d54501db6ca347819bbeeb9f3d3d4d3f"
format_stamp: "Formatted at 2026-09-24 10:35:48 on dist-test-slave-8wc7"
server_key: "1ad8c153af326b90a4050397c007fe50"
server_key_iv: "2e505d98c58eece9a02888d5ed7c2740"
server_key_version: "encryptionkey@0"
I20260924 10:35:48.303853 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "d54501db6ca347819bbeeb9f3d3d4d3f"
format_stamp: "Formatted at 2026-09-24 10:35:48 on dist-test-slave-8wc7"
server_key: "1ad8c153af326b90a4050397c007fe50"
server_key_iv: "2e505d98c58eece9a02888d5ed7c2740"
server_key_version: "encryptionkey@0"
I20260924 10:35:48.307730 13422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 10:35:48.310220 13437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:48.311601 13422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:35:48.311738 13422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "d54501db6ca347819bbeeb9f3d3d4d3f"
format_stamp: "Formatted at 2026-09-24 10:35:48 on dist-test-slave-8wc7"
server_key: "1ad8c153af326b90a4050397c007fe50"
server_key_iv: "2e505d98c58eece9a02888d5ed7c2740"
server_key_version: "encryptionkey@0"
I20260924 10:35:48.311859 13422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:35:48.353044 13422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:35:48.356122 13422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:35:48.356333 13422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:35:48.356949 13422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:35:48.357847 13422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:35:48.357894 13422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:48.357964 13422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:35:48.358004 13422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:35:48.367221 13422 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:43077
I20260924 10:35:48.367240 13550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:43077 every 8 connection(s)
I20260924 10:35:48.368189 13422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:35:48.374818 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 13422
I20260924 10:35:48.374929 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:35:48.375208 2822 external_mini_cluster.cc:1428] Setting key 30f2eb79851841ba8e2f29bdea2dd47a
Sep 24 10:35:48 dist-test-slave-8wc7 krb5kdc[13047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246148, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:35:48.382445 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.370161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:50297 (local address 127.2.193.190:40447)
0924 10:35:48.370310 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.370315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.371230 (+ 915us) server_negotiation.cc:407] Connection header received
0924 10:35:48.372552 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.372555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.372603 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.372685 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:48.373940 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.374471 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.375741 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.375965 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.376081 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.378807 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:48.378810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:48.378812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:48.378836 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:48.380264 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:48.380824 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:48.380826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:48.380828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:48.380906 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:48.381365 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:48.381367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:48.381369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:48.381644 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:48.381731 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.382209 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.382314 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:35:48.383278 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.370600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40447 (local address 127.2.193.131:50297)
0924 10:35:48.371058 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.371103 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:48.372320 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:48.372852 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:48.372860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:48.373303 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:35:48.373797 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.373811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.374591 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.374594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.375607 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.375616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.376118 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.376122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.376332 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.377051 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:35:48.377067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:35:48.378633 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:35:48.380420 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:48.380426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:48.380443 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:48.380734 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:48.381018 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:35:48.381024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:35:48.381027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:35:48.381165 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:35:48.381739 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:35:48.381744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:35:48.381983 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 10:35:48.382135 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:35:48.382387 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20260924 10:35:48.384397 13551 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40447
I20260924 10:35:48.384630 13551 heartbeater.cc:464] Registering TS with master...
I20260924 10:35:48.385149 13551 heartbeater.cc:511] Master 127.2.193.190:40447 requested a full tablet report, sending...
I20260924 10:35:48.386178 13092 ts_manager.cc:194] Registered new tserver with Master: d54501db6ca347819bbeeb9f3d3d4d3f (127.2.193.131:43077)
I20260924 10:35:48.386695 13092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:50297
I20260924 10:35:48.389124 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:35:48.397915 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.390973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.2.193.190:40447)
0924 10:35:48.391068 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.391071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.391312 (+ 241us) server_negotiation.cc:407] Connection header received
0924 10:35:48.391402 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.391405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.391439 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.391511 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:35:48.392167 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.392657 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.393514 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.393713 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.393787 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.395048 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:35:48.395051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:35:48.395053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:35:48.395074 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:35:48.396404 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:48.396774 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:48.396776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:48.396777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:48.396844 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:35:48.397062 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:35:48.397064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:35:48.397065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:35:48.397321 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:35:48.397420 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.397713 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.397804 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:35:48.400700 13092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:35:48.403708 13092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:35:48.417486 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.411591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:43077 (local address 127.0.0.1:52710)
0924 10:35:48.412380 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.412430 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:48.412568 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:48.413367 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:48.413374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:48.413401 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:48.413539 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.413546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.414591 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.414597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.415831 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.415840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.416909 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.416913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.417027 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.417064 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:35:48.417197 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:35:48.417312 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":25,"spinlock_wait_cycles":5248,"thread_start_us":57,"threads_started":1}
I20260924 10:35:48.417510 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.411747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.2.193.131:43077)
0924 10:35:48.412030 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.412042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.412713 (+ 671us) server_negotiation.cc:407] Connection header received
0924 10:35:48.412796 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.412801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.413021 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.413194 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:48.413667 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.414465 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.415959 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.416780 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.416879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.417012 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.417245 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.417354 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260924 10:35:48.417865 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.412265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:45207 (local address 127.0.0.1:32808)
0924 10:35:48.412867 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.412890 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:48.412965 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:48.413472 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:48.413475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:48.413490 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:48.413609 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.413619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.415048 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.415051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.416222 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.416230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.417230 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.417234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.417310 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.417329 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:35:48.417409 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:35:48.417461 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":112,"threads_started":1}
I20260924 10:35:48.418172 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.412226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.2.193.130:46809)
0924 10:35:48.413243 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.413257 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.413286 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:35:48.413345 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.413349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.413529 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.413672 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:48.413833 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.414921 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.416414 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.417276 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.417372 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.417487 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.417941 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.418045 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":144,"threads_started":1}
I20260924 10:35:48.418200 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.411599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46809 (local address 127.0.0.1:42430)
0924 10:35:48.412388 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.412435 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:48.412569 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:48.413656 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:48.413659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:48.413674 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:48.413749 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.413754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.415106 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.415111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.416268 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.416281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.417553 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.417557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.417652 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.417676 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:35:48.417754 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:35:48.417822 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":2,"spinlock_wait_cycles":4608,"thread_start_us":98,"threads_started":1}
I20260924 10:35:48.418509 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.412698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.2.193.129:45207)
0924 10:35:48.413021 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.413028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.413048 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:35:48.413121 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.413126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.413342 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.413492 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:48.413739 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.414919 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.416319 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.416924 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.417023 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.417153 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.417888 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.418140 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":68,"threads_started":1}
I20260924 10:35:48.420271 13485 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5a4f9ca82149c790bbf2fa711c2443 (DEFAULT_TABLE table=test-table [id=10ba300fd1df440fbf61cfd5fee79d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:48.420818 13217 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5a4f9ca82149c790bbf2fa711c2443 (DEFAULT_TABLE table=test-table [id=10ba300fd1df440fbf61cfd5fee79d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:48.421236 13485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5a4f9ca82149c790bbf2fa711c2443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:48.421700 13217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5a4f9ca82149c790bbf2fa711c2443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:48.423653 13351 tablet_service.cc:1467] Processing CreateTablet for tablet 7d5a4f9ca82149c790bbf2fa711c2443 (DEFAULT_TABLE table=test-table [id=10ba300fd1df440fbf61cfd5fee79d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:35:48.424610 13351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d5a4f9ca82149c790bbf2fa711c2443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:35:48.428484 13568 tablet_bootstrap.cc:492] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Bootstrap starting.
I20260924 10:35:48.429340 13569 tablet_bootstrap.cc:492] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b: Bootstrap starting.
I20260924 10:35:48.431365 13568 tablet_bootstrap.cc:654] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:48.431479 13569 tablet_bootstrap.cc:654] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:48.431530 13570 tablet_bootstrap.cc:492] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Bootstrap starting.
I20260924 10:35:48.432332 13568 log.cc:824] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:48.432332 13569 log.cc:824] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:48.434010 13570 tablet_bootstrap.cc:654] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Neither blocks nor log segments found. Creating new log.
I20260924 10:35:48.434610 13569 tablet_bootstrap.cc:492] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b: No bootstrap required, opened a new log
I20260924 10:35:48.434717 13568 tablet_bootstrap.cc:492] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: No bootstrap required, opened a new log
I20260924 10:35:48.434791 13570 log.cc:824] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Log is configured to *not* fsync() on all Append() calls
I20260924 10:35:48.434819 13569 ts_tablet_manager.cc:1397] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:35:48.434892 13568 ts_tablet_manager.cc:1397] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:35:48.436659 13570 tablet_bootstrap.cc:492] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: No bootstrap required, opened a new log
I20260924 10:35:48.436836 13570 ts_tablet_manager.cc:1397] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:35:48.438367 13569 raft_consensus.cc:348] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.438612 13569 raft_consensus.cc:374] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:48.438663 13569 raft_consensus.cc:729] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f7b006a0a948b792c2ddebb745f50b, State: Initialized, Role: FOLLOWER
I20260924 10:35:48.438630 13568 raft_consensus.cc:348] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.438871 13568 raft_consensus.cc:374] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:48.438925 13568 raft_consensus.cc:729] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d54501db6ca347819bbeeb9f3d3d4d3f, State: Initialized, Role: FOLLOWER
I20260924 10:35:48.439349 13569 consensus_queue.cc:260] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.439333 13568 consensus_queue.cc:260] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.440326 13568 ts_tablet_manager.cc:1428] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 10:35:48.440539 13570 raft_consensus.cc:348] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.440765 13551 heartbeater.cc:503] Master 127.2.193.190:40447 was elected leader, sending a full tablet report...
I20260924 10:35:48.440764 13283 heartbeater.cc:503] Master 127.2.193.190:40447 was elected leader, sending a full tablet report...
I20260924 10:35:48.440780 13570 raft_consensus.cc:374] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:35:48.440820 13569 ts_tablet_manager.cc:1428] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:35:48.440825 13570 raft_consensus.cc:729] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e30f24383a94f5091ac4c432d3ebc05, State: Initialized, Role: FOLLOWER
I20260924 10:35:48.441270 13570 consensus_queue.cc:260] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.441864 13417 heartbeater.cc:503] Master 127.2.193.190:40447 was elected leader, sending a full tablet report...
I20260924 10:35:48.442128 13570 ts_tablet_manager.cc:1428] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:35:48.469655 13575 raft_consensus.cc:482] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:35:48.469902 13575 raft_consensus.cc:504] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.471134 13575 leader_election.cc:288] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d54501db6ca347819bbeeb9f3d3d4d3f (127.2.193.131:43077), 0e30f24383a94f5091ac4c432d3ebc05 (127.2.193.130:46809)
I20260924 10:35:48.475383 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.471629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:57753 (local address 127.2.193.130:46809)
0924 10:35:48.471760 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.471765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.471972 (+ 207us) server_negotiation.cc:407] Connection header received
0924 10:35:48.472119 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.472125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.472177 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.472250 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:48.472556 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.473126 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.474395 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.474804 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.474942 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.474996 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.475161 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.475239 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:35:48.475391 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.471528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:57251 (local address 127.2.193.131:43077)
0924 10:35:48.471678 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.471683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.471963 (+ 280us) server_negotiation.cc:407] Connection header received
0924 10:35:48.472213 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.472217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.472292 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.472370 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:35:48.472751 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.473302 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.474427 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.474812 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.474942 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.474992 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.475216 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.475288 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:35:48.475415 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.471425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46809 (local address 127.2.193.129:57753)
0924 10:35:48.471858 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.471875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:48.471999 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:48.472280 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:48.472284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:48.472300 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:48.472449 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.472459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.473302 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.473306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.474266 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.474276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.474930 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.474935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.475037 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.475063 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:35:48.475123 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:35:48.475210 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":53,"thread_start_us":123,"threads_started":1}
I20260924 10:35:48.475407 13578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.471425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:43077 (local address 127.2.193.129:57251)
0924 10:35:48.471881 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.471895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:35:48.472039 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:35:48.472424 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:35:48.472427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:35:48.472446 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:35:48.472571 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.472577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.473418 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.473420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.474267 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:35:48.474278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.474957 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.474961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:35:48.475077 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.475094 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:35:48.475190 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:35:48.475253 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":214,"thread_start_us":122,"threads_started":1}
I20260924 10:35:48.476227 13371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5a4f9ca82149c790bbf2fa711c2443" candidate_uuid: "d9f7b006a0a948b792c2ddebb745f50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e30f24383a94f5091ac4c432d3ebc05" is_pre_election: true
I20260924 10:35:48.476233 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5a4f9ca82149c790bbf2fa711c2443" candidate_uuid: "d9f7b006a0a948b792c2ddebb745f50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" is_pre_election: true
I20260924 10:35:48.476572 13505 raft_consensus.cc:2454] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9f7b006a0a948b792c2ddebb745f50b in term 0.
I20260924 10:35:48.476572 13371 raft_consensus.cc:2454] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9f7b006a0a948b792c2ddebb745f50b in term 0.
I20260924 10:35:48.477095 13171 leader_election.cc:302] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e30f24383a94f5091ac4c432d3ebc05, d54501db6ca347819bbeeb9f3d3d4d3f, d9f7b006a0a948b792c2ddebb745f50b; no voters:
I20260924 10:35:48.477421 13575 raft_consensus.cc:2781] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:35:48.477509 13575 raft_consensus.cc:482] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:35:48.477558 13575 raft_consensus.cc:3037] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:48.478639 13575 raft_consensus.cc:504] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.478981 13575 leader_election.cc:288] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [CANDIDATE]: Term 1 election: Requested vote from peers d54501db6ca347819bbeeb9f3d3d4d3f (127.2.193.131:43077), 0e30f24383a94f5091ac4c432d3ebc05 (127.2.193.130:46809)
I20260924 10:35:48.479393 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5a4f9ca82149c790bbf2fa711c2443" candidate_uuid: "d9f7b006a0a948b792c2ddebb745f50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f"
I20260924 10:35:48.479434 13371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d5a4f9ca82149c790bbf2fa711c2443" candidate_uuid: "d9f7b006a0a948b792c2ddebb745f50b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e30f24383a94f5091ac4c432d3ebc05"
I20260924 10:35:48.479568 13371 raft_consensus.cc:3037] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:48.479578 13505 raft_consensus.cc:3037] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:35:48.480672 13371 raft_consensus.cc:2454] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9f7b006a0a948b792c2ddebb745f50b in term 1.
I20260924 10:35:48.480865 13505 raft_consensus.cc:2454] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9f7b006a0a948b792c2ddebb745f50b in term 1.
I20260924 10:35:48.480965 13170 leader_election.cc:302] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e30f24383a94f5091ac4c432d3ebc05, d9f7b006a0a948b792c2ddebb745f50b; no voters:
I20260924 10:35:48.481387 13575 raft_consensus.cc:2781] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:35:48.481633 13575 raft_consensus.cc:686] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 1 LEADER]: Becoming Leader. State: Replica: d9f7b006a0a948b792c2ddebb745f50b, State: Running, Role: LEADER
I20260924 10:35:48.481968 13575 consensus_queue.cc:237] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } }
I20260924 10:35:48.485351 13091 catalog_manager.cc:5495] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b reported cstate change: term changed from 0 to 1, leader changed from <none> to d9f7b006a0a948b792c2ddebb745f50b (127.2.193.129). New cstate: current_term: 1 leader_uuid: "d9f7b006a0a948b792c2ddebb745f50b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9f7b006a0a948b792c2ddebb745f50b" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45207 } health_report { overall_health: HEALTHY } } }
I20260924 10:35:48.495658 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.492383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.2.193.129:45207)
0924 10:35:48.492509 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:35:48.492513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:35:48.492583 (+ 70us) server_negotiation.cc:407] Connection header received
0924 10:35:48.492710 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:35:48.492712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:35:48.492762 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:35:48.492846 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:35:48.493058 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.493602 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.494425 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:35:48.494627 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:35:48.494698 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:35:48.494957 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:35:48.495357 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:35:48.495443 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:35:48.495505 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 10:35:48.495553 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:35:48.501446 13505 raft_consensus.cc:1261] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 1 FOLLOWER]: Refusing update from remote peer d9f7b006a0a948b792c2ddebb745f50b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:48.501446 13371 raft_consensus.cc:1261] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 1 FOLLOWER]: Refusing update from remote peer d9f7b006a0a948b792c2ddebb745f50b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:35:48.502169 13579 consensus_queue.cc:1035] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d54501db6ca347819bbeeb9f3d3d4d3f" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:48.502352 13575 consensus_queue.cc:1035] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e30f24383a94f5091ac4c432d3ebc05" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:35:48.509023 13582 mvcc.cc:205] Tried to move back new op lower bound from 7332848224254767104 to 7332848224185090048. Current Snapshot: MvccSnapshot[applied={T|T < 7332848224254767104}]
I20260924 10:35:48.511181 13583 mvcc.cc:205] Tried to move back new op lower bound from 7332848224254767104 to 7332848224185090048. Current Snapshot: MvccSnapshot[applied={T|T < 7332848224254767104}]
I20260924 10:35:48.518133 13091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:48.518326 13091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:35:48.520751 13091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b22724ce064e64894dddc2aa37aa3a: Sending DeleteTablet for 3 replicas of tablet 7d5a4f9ca82149c790bbf2fa711c2443
I20260924 10:35:48.521556 13485 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5a4f9ca82149c790bbf2fa711c2443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:48 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52710
I20260924 10:35:48.521678 13351 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5a4f9ca82149c790bbf2fa711c2443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:48 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:42430
I20260924 10:35:48.521688 13217 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d5a4f9ca82149c790bbf2fa711c2443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:35:48 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:32808
I20260924 10:35:48.522159 13594 tablet_replica.cc:330] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b: stopping tablet replica
I20260924 10:35:48.522166 13592 tablet_replica.cc:330] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: stopping tablet replica
I20260924 10:35:48.522465 13594 raft_consensus.cc:2229] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:35:48.522476 13592 raft_consensus.cc:2229] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:48.522792 13592 raft_consensus.cc:2258] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:48.522805 13594 raft_consensus.cc:2258] T 7d5a4f9ca82149c790bbf2fa711c2443 P d9f7b006a0a948b792c2ddebb745f50b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:48.523250 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 13154
I20260924 10:35:48.525410 13593 tablet_replica.cc:330] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: stopping tablet replica
I20260924 10:35:48.525484 13592 ts_tablet_manager.cc:1905] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:48.525733 13593 raft_consensus.cc:2229] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:35:48.526060 13593 raft_consensus.cc:2258] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:35:48.527592 13593 ts_tablet_manager.cc:1905] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:35:48.528415 13592 ts_tablet_manager.cc:1918] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:48.528517 13592 log.cc:1196] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/7d5a4f9ca82149c790bbf2fa711c2443
I20260924 10:35:48.528849 13592 ts_tablet_manager.cc:1939] T 7d5a4f9ca82149c790bbf2fa711c2443 P d54501db6ca347819bbeeb9f3d3d4d3f: Deleting consensus metadata
I20260924 10:35:48.530242 13077 catalog_manager.cc:4841] TS d54501db6ca347819bbeeb9f3d3d4d3f (127.2.193.131:43077): tablet 7d5a4f9ca82149c790bbf2fa711c2443 (table test-table [id=10ba300fd1df440fbf61cfd5fee79d1d]) successfully deleted
I20260924 10:35:48.530433 13593 ts_tablet_manager.cc:1918] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:35:48.530532 13593 log.cc:1196] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/7d5a4f9ca82149c790bbf2fa711c2443
W20260924 10:35:48.530757 13076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:45207 (error 108)
I20260924 10:35:48.530866 13593 ts_tablet_manager.cc:1939] T 7d5a4f9ca82149c790bbf2fa711c2443 P 0e30f24383a94f5091ac4c432d3ebc05: Deleting consensus metadata
I20260924 10:35:48.530980 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 13288
I20260924 10:35:48.533052 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.532663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:45207 (local address 127.0.0.1:32836)
0924 10:35:48.532863 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.532945 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:45207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 10:35:48.533252 13076 catalog_manager.cc:4584] TS d9f7b006a0a948b792c2ddebb745f50b (127.2.193.129:45207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d5a4f9ca82149c790bbf2fa711c2443: Network error: Client connection negotiation failed: client connection to 127.2.193.129:45207: connect: Connection refused (error 111)
I20260924 10:35:48.536366 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 13422
I20260924 10:35:48.537453 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:35:48.536503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46809 (local address 127.0.0.1:42432)
0924 10:35:48.537334 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 10:35:48.537384 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:46809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":754}
I20260924 10:35:48.541733 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 13060
2026-09-24T10:35:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2285 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:35:48 dist-test-slave-8wc7 krb5kdc[13598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:35:48 dist-test-slave-8wc7 krb5kdc[13598](info): set up 2 sockets
Sep 24 10:35:48 dist-test-slave-8wc7 krb5kdc[13598](info): commencing operation
krb5kdc: starting...
W20260924 10:35:49.582186 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:35:49 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:35:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/rangeradmin_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/rangeradmin_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/rangerlookup_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/rangerlookup_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
I20260924 10:35:49.752390 2822 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBKECxY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/postgres -l logfile start
2026-09-24 10:35:52.196 UTC [13628] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:35:52.196 UTC [13628] LOG: listening on IPv4 address "127.2.193.148", port 58557
2026-09-24 10:35:52.200 UTC [13628] LOG: listening on Unix socket "/tmp/.s.PGSQL.58557"
2026-09-24 10:35:52.234 UTC [13631] LOG: database system was shut down at 2026-09-24 10:35:50 UTC
2026-09-24 10:35:52.238 UTC [13628] LOG: database system is ready to accept connections
I20260924 10:35:53.190124 2822 mini_postgres.cc:96] Postgres bound to 58557
2026-09-24 10:35:53.196 UTC [13639] FATAL: database "slave" does not exist
127.2.193.148:58557 - accepting connections
I20260924 10:35:53.197587 2822 mini_ranger.cc:161] Starting Ranger...
I20260924 10:35:53.213230 2822 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:35:53.699254 2822 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:35:53.699368 2822 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-admin
2026-09-24 10:35:53,976 [I] DB FLAVOR :POSTGRES
2026-09-24 10:35:53,976 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:35:53,976 [I] Checking connection..
2026-09-24 10:35:53,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:35:54,224 [I] Checking connection passed.
2026-09-24 10:35:54,225 [I] --------- Verifying version history table ---------
2026-09-24 10:35:54,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:35:54,499 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:35:54,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:35:54,745 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:35:54,745 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:35:54,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:35:54.984 UTC [13703] WARNING: there is no transaction in progress
2026-09-24 10:35:54,990 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:35:54,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:35:55,226 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:35:55,226 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:35:55,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:35:55,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:35:55,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8wc7.c.gcp-upstream.internal','N') ;"
2026-09-24 10:35:55,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:35:56,223 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:35:56,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:35:56,474 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:35:56,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:35:56,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:35:56,985 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:35:56,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:35:57.972 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:57.988 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.004 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.019 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.039 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.056 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.111 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.117 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.130 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.140 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.150 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.162 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.172 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.179 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.186 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58.194 UTC [13811] WARNING: there is no transaction in progress
2026-09-24 10:35:58,813 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:35:58,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:35:59,044 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:35:59,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:35:59,281 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:35:59,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:35:59,522 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:35:59,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:35:59,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:36:00,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8wc7.c.gcp-upstream.internal';"
2026-09-24 10:36:00,296 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:36:00,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:36:00,552 [I] DB_PATCHES have already been applied
I20260924 10:36:00.558741 2822 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBKECxY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:36:00.558810 2822 mini_ranger.cc:193] Using host: 127.2.193.148
I20260924 10:36:00.562424 2822 mini_ranger.cc:227] Ranger admin URL: http://127.2.193.148:43565
Sep 24, 2026 10:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:36:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:36:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:36:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:36:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:36:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:36:01.693051 2822 mini_ranger.cc:160] Time spent starting Ranger: real 8.496s user 0.001s sys 0.010s
Sep 24, 2026 10:36:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:36:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 10:36:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:36:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:36:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:36:23.360260 2822 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/rangerkms_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/rangerkms_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.193.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.193.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:36:23 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246183, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:36:23.452191 2822 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:36:23.461712 2822 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:36:24.286482 2822 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:36:24.286582 2822 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms
I20260924 10:36:24.286693 2822 mini_ranger_kms.cc:227] Using postgres at 127.2.193.148:58557
2026-09-24 10:36:24,483 [I] DB FLAVOR :POSTGRES
2026-09-24 10:36:24,483 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:36:24,483 [I] Checking connection
2026-09-24 10:36:24,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:36:24,747 [I] connection success
2026-09-24 10:36:24,747 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:36:24,748 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:36:24,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:36:24,989 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:36:24,989 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:36:24,989 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:36:24,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:58557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:36:25,245 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:36:25.732125 2822 mini_ranger_kms.cc:310] Created kms service
I20260924 10:36:25.912467 2822 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:36:25.959079 2822 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:36:25 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246183, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:36:26.366782 2822 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:36:26.367125 2822 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:36:26.367162 2822 mini_ranger_kms.cc:240] Using host: 127.2.193.148
I20260924 10:36:26.371044 2822 mini_ranger_kms.cc:276] Ranger KMS PID: 13991
I20260924 10:36:26.371191 2822 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.193.148:39521
Sep 24, 2026 10:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:36:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 10:36:28 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246188, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.193.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:36:28.637354 2822 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.185s user 0.006s sys 0.173s
I20260924 10:36:28.637501 2822 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:36:28.637617 2822 mini_ranger_kms.cc:401] 127.2.193.148:39521/kms/v1/keys
Sep 24 10:36:28 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246188, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.193.148@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
Sep 24 10:36:29 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246188, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.193.148@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
Sep 24 10:36:31 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:36:31.724306 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34691
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:34691
--ranger_config_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:39521/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:36:31.859696 14042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:36:31.859997 14042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:36:31.860038 14042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:36:31.864351 14042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:36:31.864427 14042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:36:31.864447 14042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:36:31.864465 14042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:36:31.864481 14042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:36:31.870316 14042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.193.148:39521/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:34691
--ranger_config_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:36:31.871486 14042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:36:31.872574 14042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:36:31.882694 14047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:36:31.882789 14042 server_base.cc:1034] running on GCE node
W20260924 10:36:31.882725 14050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:36:31.883013 14048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:36:31.883737 14042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:36:31.885280 14042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:36:31.886493 14042 hybrid_clock.cc:648] HybridClock initialized: now 1790246191886461 us; error 36 us; skew 500 ppm
Sep 24 10:36:31 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246191, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:36:31.889518 14042 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:36:31.890465 14042 webserver.cc:460] Webserver started at http://127.2.193.190:45133/ using document root <none> and password file <none>
I20260924 10:36:31.891058 14042 fs_manager.cc:359] Metadata directory not provided
I20260924 10:36:31.891141 14042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:36:31.891362 14042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:36:31 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246191, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:36:31.951061 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "6bf3836fa51c4807aa0acb6289abf8f1"
format_stamp: "Formatted at 2026-09-24 10:36:31 on dist-test-slave-8wc7"
server_key: "ab036476f94192e882e2ae8d3790e32e"
server_key_iv: "af7c86f8967c2eed2d64b80b67efed82"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:31.951817 14042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "6bf3836fa51c4807aa0acb6289abf8f1"
format_stamp: "Formatted at 2026-09-24 10:36:31 on dist-test-slave-8wc7"
server_key: "ab036476f94192e882e2ae8d3790e32e"
server_key_iv: "af7c86f8967c2eed2d64b80b67efed82"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:31.956862 14042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 10:36:31.972224 14057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:31.973861 14042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:36:31.974042 14042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "6bf3836fa51c4807aa0acb6289abf8f1"
format_stamp: "Formatted at 2026-09-24 10:36:31 on dist-test-slave-8wc7"
server_key: "ab036476f94192e882e2ae8d3790e32e"
server_key_iv: "af7c86f8967c2eed2d64b80b67efed82"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:31.974187 14042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:36:31.985394 14042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:36:31.988469 14042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:36:31.988688 14042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:36:31.997215 14042 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:34691
I20260924 10:36:31.997236 14109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:34691 every 8 connection(s)
I20260924 10:36:31.998263 14042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:36:32.001111 14110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:36:32.003424 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 14042
I20260924 10:36:32.003564 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:36:32.003881 2822 external_mini_cluster.cc:1428] Setting key 81294e5cd36bb8c2a8c884a71dbac904
I20260924 10:36:32.007768 14110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1: Bootstrap starting.
I20260924 10:36:32.010592 14110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1: Neither blocks nor log segments found. Creating new log.
I20260924 10:36:32.011415 14110 log.cc:824] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:36:32.013703 14110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1: No bootstrap required, opened a new log
Sep 24 10:36:32 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:36:32.018096 14110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34691 } }
I20260924 10:36:32.018426 14110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:36:32.018520 14110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf3836fa51c4807aa0acb6289abf8f1, State: Initialized, Role: FOLLOWER
I20260924 10:36:32.019049 14110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34691 } }
I20260924 10:36:32.019253 14110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:36:32.019354 14110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:36:32.019475 14110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:36:32.020602 14110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34691 } }
I20260924 10:36:32.020818 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:32.005450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.2.193.190:34691)
0924 10:36:32.005928 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:32.005947 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:32.005993 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:36:32.006848 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:32.006873 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:32.007320 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:32.007702 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:36:32.008624 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:32.010124 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:32.011417 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:32.012068 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:32.012262 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:32.015652 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:36:32.015662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:36:32.015680 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:36:32.015714 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:36:32.017651 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:32.018086 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:32.018092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:32.018097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:32.018226 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:32.018492 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:32.018495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:32.018498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:32.019011 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:36:32.019198 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:32.019539 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 10:36:32.019829 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":143,"threads_started":1}
I20260924 10:36:32.020962 14110 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf3836fa51c4807aa0acb6289abf8f1; no voters:
I20260924 10:36:32.022163 14110 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:36:32.022270 14115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:36:32.022480 14115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf3836fa51c4807aa0acb6289abf8f1, State: Running, Role: LEADER
I20260924 10:36:32.022845 14115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34691 } }
I20260924 10:36:32.023507 14110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:36:32.024106 14115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf3836fa51c4807aa0acb6289abf8f1. Latest consensus state: current_term: 1 leader_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34691 } } }
I20260924 10:36:32.024302 14115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:36:32.024560 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf3836fa51c4807aa0acb6289abf8f1" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34691 } } }
I20260924 10:36:32.024693 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1 [sys.catalog]: This master's current role is: LEADER
I20260924 10:36:32.024791 14123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:36:32.028039 14123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:36:32.028446 14110 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:36:32.038553 14123 catalog_manager.cc:1347] Generated new cluster ID: dd4b2a14278943e3ae5762b85587559b
I20260924 10:36:32.038774 14123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:36:32.052636 14123 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:36:32.054107 14123 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:36:32.070766 14123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1: Generated new TSK 0
I20260924 10:36:32.071785 14123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:36:32.780186 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:39521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34691
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:36:32.941692 14155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:36:32.942011 14155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:36:32.942090 14155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:36:32.947607 14155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:36:32.947772 14155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:36:32.947938 14155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:36:32.956293 14155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.193.148:39521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:36:32.957806 14155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:36:32.959295 14155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:36:32.971969 14163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:36:32.971980 14165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:36:32.971987 14162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:36:32.973061 14155 server_base.cc:1034] running on GCE node
I20260924 10:36:32.973489 14155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:36:32.974220 14155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:36:32.975394 14155 hybrid_clock.cc:648] HybridClock initialized: now 1790246192975355 us; error 54 us; skew 500 ppm
Sep 24 10:36:32 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:36:32.978791 14155 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:36:32.979976 14155 webserver.cc:460] Webserver started at http://127.2.193.129:35269/ using document root <none> and password file <none>
I20260924 10:36:32.980758 14155 fs_manager.cc:359] Metadata directory not provided
I20260924 10:36:32.980854 14155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:36:32.981128 14155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:36:32 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:36:33.002537 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15"
format_stamp: "Formatted at 2026-09-24 10:36:32 on dist-test-slave-8wc7"
server_key: "290c24b49f1d99ecc392728993c8531c"
server_key_iv: "9401bd13192470349a87e8b6ad36a897"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.003381 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15"
format_stamp: "Formatted at 2026-09-24 10:36:32 on dist-test-slave-8wc7"
server_key: "290c24b49f1d99ecc392728993c8531c"
server_key_iv: "9401bd13192470349a87e8b6ad36a897"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.009631 14155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 10:36:33.039072 14172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.041109 14155 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260924 10:36:33.041312 14155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15"
format_stamp: "Formatted at 2026-09-24 10:36:32 on dist-test-slave-8wc7"
server_key: "290c24b49f1d99ecc392728993c8531c"
server_key_iv: "9401bd13192470349a87e8b6ad36a897"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.041467 14155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:36:33.077091 14155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:36:33.081401 14155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:36:33.081648 14155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:36:33.082430 14155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:36:33.083595 14155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:36:33.083700 14155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.083786 14155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:36:33.083837 14155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.097196 14287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:44975 every 8 connection(s)
I20260924 10:36:33.097126 14155 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:44975
I20260924 10:36:33.099617 14155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:36:33.100356 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 14155
I20260924 10:36:33.100493 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:36:33.100921 2822 external_mini_cluster.cc:1428] Setting key 03260e9eb537b3c6e9b858a3b9e27936
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:36:33.128997 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.104490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34691 (local address 127.2.193.129:52887)
0924 10:36:33.105742 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:33.105800 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:33.106835 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:33.112803 (+ 5968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:33.112816 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:33.113305 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:36:33.114085 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.114107 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.115449 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.115453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.116960 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.116973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.117717 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.117722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.117938 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.120473 (+ 2535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:36:33.120498 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:36:33.122422 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:36:33.125259 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:36:33.125267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:36:33.125289 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:36:33.125712 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:36:33.126112 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:36:33.126115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:36:33.126117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:36:33.126315 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:36:33.126817 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:36:33.126824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:36:33.127204 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 10:36:33.127446 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 10:36:33.127828 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":147,"threads_started":1}
I20260924 10:36:33.130204 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.103990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:52887 (local address 127.2.193.190:34691)
0924 10:36:33.112200 (+ 8210us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:33.112205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:33.112226 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:36:33.112281 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:33.112285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:33.112347 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:33.114270 (+ 1923us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:36:33.114410 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.115267 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.117137 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.117524 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.117644 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.122674 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:36:33.122679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:36:33.122682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:36:33.122715 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:36:33.125022 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.125894 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.125897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.125899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.125997 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.126440 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.126443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.126445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.126688 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:36:33.129764 (+ 3076us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:33.129875 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 10:36:33.130026 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8103,"thread_start_us":83,"threads_started":1}
I20260924 10:36:33.131531 14288 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34691
I20260924 10:36:33.131812 14288 heartbeater.cc:464] Registering TS with master...
I20260924 10:36:33.132476 14288 heartbeater.cc:511] Master 127.2.193.190:34691 requested a full tablet report, sending...
I20260924 10:36:33.134536 14074 ts_manager.cc:194] Registered new tserver with Master: e81d4ddbb8244cf5b8c893d38d0c0f15 (127.2.193.129:44975)
I20260924 10:36:33.136212 14074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:52887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:36:33.141948 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:39521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34691
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:36:33.311830 14294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:36:33.312333 14294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:36:33.312502 14294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:36:33.316847 14294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:36:33.316955 14294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:36:33.317080 14294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:36:33.323210 14294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.193.148:39521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:36:33.324555 14294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:36:33.325781 14294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:36:33.336019 14303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:36:33.336149 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:36:33.336038 14300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:36:33.337422 14294 server_base.cc:1034] running on GCE node
I20260924 10:36:33.337846 14294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:36:33.338550 14294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:36:33.339766 14294 hybrid_clock.cc:648] HybridClock initialized: now 1790246193339729 us; error 48 us; skew 500 ppm
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:36:33.342700 14294 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:36:33.343742 14294 webserver.cc:460] Webserver started at http://127.2.193.130:39483/ using document root <none> and password file <none>
I20260924 10:36:33.344318 14294 fs_manager.cc:359] Metadata directory not provided
I20260924 10:36:33.344419 14294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:36:33.344641 14294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:36:33.371402 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "04d60462f5944c6e80e3e396ecbad6ac"
format_stamp: "Formatted at 2026-09-24 10:36:33 on dist-test-slave-8wc7"
server_key: "b839b7675ed293da94bba086d2c65a0b"
server_key_iv: "b1701b54bd709d90a8ae59862f8b07d7"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.372138 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "04d60462f5944c6e80e3e396ecbad6ac"
format_stamp: "Formatted at 2026-09-24 10:36:33 on dist-test-slave-8wc7"
server_key: "b839b7675ed293da94bba086d2c65a0b"
server_key_iv: "b1701b54bd709d90a8ae59862f8b07d7"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.382943 14294 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.000s
I20260924 10:36:33.402911 14310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.404738 14294 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260924 10:36:33.404896 14294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "04d60462f5944c6e80e3e396ecbad6ac"
format_stamp: "Formatted at 2026-09-24 10:36:33 on dist-test-slave-8wc7"
server_key: "b839b7675ed293da94bba086d2c65a0b"
server_key_iv: "b1701b54bd709d90a8ae59862f8b07d7"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.405063 14294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:36:33.422688 14294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:36:33.427858 14294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:36:33.428078 14294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:36:33.440640 14294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:36:33.442746 14294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:36:33.442832 14294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.442891 14294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:36:33.442922 14294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.454737 14294 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:46859
I20260924 10:36:33.454789 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:46859 every 8 connection(s)
I20260924 10:36:33.456007 14294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:36:33.464820 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 14294
I20260924 10:36:33.464957 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:36:33.465767 2822 external_mini_cluster.cc:1428] Setting key 92139d4d74f8b9f0be918aacf8ec7021
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:36:33.488238 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.465321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:50317 (local address 127.2.193.190:34691)
0924 10:36:33.465468 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:33.465473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:33.469726 (+ 4253us) server_negotiation.cc:407] Connection header received
0924 10:36:33.471124 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:33.471128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:33.471188 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:33.471246 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:36:33.473159 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.474031 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.476340 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.476727 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.476817 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.483463 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:36:33.483468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:36:33.483471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:36:33.483520 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:36:33.485385 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.486127 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.486132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.486134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.486228 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.486722 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.486726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.486728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.486979 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:36:33.487059 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:33.487903 (+ 844us) server_negotiation.cc:299] Negotiation successful
0924 10:36:33.488062 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:36:33.494589 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.463339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34691 (local address 127.2.193.130:50317)
0924 10:36:33.466598 (+ 3259us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:33.469608 (+ 3010us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:33.470816 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:33.471749 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:33.471760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:33.472163 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:36:33.472952 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.472977 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.474205 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.474208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.476182 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.476194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.477057 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.477061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.477311 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.478668 (+ 1357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:36:33.478691 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:36:33.483187 (+ 4496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:36:33.485562 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:36:33.485569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:36:33.485590 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:36:33.485989 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:36:33.486353 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:36:33.486356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:36:33.486359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:36:33.486594 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:36:33.487147 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:36:33.487154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:36:33.487585 (+ 431us) client_negotiation.cc:769] Sending connection context
0924 10:36:33.488783 (+ 1198us) client_negotiation.cc:240] Negotiation successful
0924 10:36:33.493336 (+ 4553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2990,"thread_start_us":173,"threads_started":1}
I20260924 10:36:33.496486 14424 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34691
I20260924 10:36:33.496767 14424 heartbeater.cc:464] Registering TS with master...
I20260924 10:36:33.497675 14424 heartbeater.cc:511] Master 127.2.193.190:34691 requested a full tablet report, sending...
I20260924 10:36:33.499015 14074 ts_manager.cc:194] Registered new tserver with Master: 04d60462f5944c6e80e3e396ecbad6ac (127.2.193.130:46859)
I20260924 10:36:33.499827 14074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:50317
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:36:33.530593 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:39521/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34691
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
W20260924 10:36:33.689975 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:36:33.690416 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:36:33.690508 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:36:33.696853 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:36:33.696960 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:36:33.697064 14429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:36:33.705258 14429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.193.148:39521/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:36:33.706722 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:36:33.707891 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:36:33.718273 14437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:36:33.718417 14429 server_base.cc:1034] running on GCE node
W20260924 10:36:33.718297 14435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:36:33.719074 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:36:33.719646 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:36:33.720439 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:36:33.721643 14429 hybrid_clock.cc:648] HybridClock initialized: now 1790246193721600 us; error 61 us; skew 500 ppm
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:36:33.725294 14429 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:36:33.726423 14429 webserver.cc:460] Webserver started at http://127.2.193.131:45909/ using document root <none> and password file <none>
I20260924 10:36:33.727195 14429 fs_manager.cc:359] Metadata directory not provided
I20260924 10:36:33.727279 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:36:33.727552 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:36:33.742684 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "54bde9b2fcf2477689e0863cebf90803"
format_stamp: "Formatted at 2026-09-24 10:36:33 on dist-test-slave-8wc7"
server_key: "d8f13ba0baa3fffb83529334a1f90fcd"
server_key_iv: "746ed8809c57d06a88f3decd15a90f86"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.743443 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "54bde9b2fcf2477689e0863cebf90803"
format_stamp: "Formatted at 2026-09-24 10:36:33 on dist-test-slave-8wc7"
server_key: "d8f13ba0baa3fffb83529334a1f90fcd"
server_key_iv: "746ed8809c57d06a88f3decd15a90f86"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.749080 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 10:36:33.767750 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.769528 14429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:36:33.769691 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "54bde9b2fcf2477689e0863cebf90803"
format_stamp: "Formatted at 2026-09-24 10:36:33 on dist-test-slave-8wc7"
server_key: "d8f13ba0baa3fffb83529334a1f90fcd"
server_key_iv: "746ed8809c57d06a88f3decd15a90f86"
server_key_version: "kuduclusterkey@0"
I20260924 10:36:33.769852 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:36:33.791620 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:36:33.795691 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:36:33.795957 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:36:33.796741 14429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:36:33.798319 14429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:36:33.798427 14429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.798524 14429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:36:33.798580 14429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:36:33.810125 14429 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:37881
I20260924 10:36:33.811704 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:36:33.812260 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 14429
I20260924 10:36:33.812384 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:36:33.812752 2822 external_mini_cluster.cc:1428] Setting key f2db118a9089d5d1a978b91e8bd325e7
I20260924 10:36:33.819120 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:37881 every 8 connection(s)
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:36:33.839135 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.821473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:35847 (local address 127.2.193.190:34691)
0924 10:36:33.821621 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:33.821625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:33.823522 (+ 1897us) server_negotiation.cc:407] Connection header received
0924 10:36:33.825008 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:33.825013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:33.825081 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:33.825147 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:36:33.827686 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.828517 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.830388 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.830742 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.830840 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.834219 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:36:33.834223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:36:33.834226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:36:33.834257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:36:33.836166 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.836999 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.837003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.837005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.837107 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.837657 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.837661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.837663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.837934 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:36:33.838039 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:33.838819 (+ 780us) server_negotiation.cc:299] Negotiation successful
0924 10:36:33.838949 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:36:33.840883 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.822052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34691 (local address 127.2.193.131:35847)
0924 10:36:33.823339 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:33.823407 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:33.824641 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:33.825829 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:33.825841 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:33.826712 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:36:33.827502 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.827528 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.828691 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.828695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.830226 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.830240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.831086 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.831091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.831306 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.832108 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:36:33.832130 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:36:33.833989 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:36:33.836377 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:36:33.836384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:36:33.836404 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:36:33.836821 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:36:33.837272 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:36:33.837279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:36:33.837282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:36:33.837519 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:36:33.838116 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:36:33.838123 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:36:33.838512 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 10:36:33.839325 (+ 813us) client_negotiation.cc:240] Negotiation successful
0924 10:36:33.839722 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":161,"threads_started":1}
I20260924 10:36:33.842203 14558 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34691
I20260924 10:36:33.842476 14558 heartbeater.cc:464] Registering TS with master...
I20260924 10:36:33.843055 14558 heartbeater.cc:511] Master 127.2.193.190:34691 requested a full tablet report, sending...
I20260924 10:36:33.844264 14074 ts_manager.cc:194] Registered new tserver with Master: 54bde9b2fcf2477689e0863cebf90803 (127.2.193.131:37881)
I20260924 10:36:33.845064 14074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:35847
I20260924 10:36:33.851291 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:36:33.863418 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.853801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.2.193.190:34691)
0924 10:36:33.853929 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:33.853933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:33.854227 (+ 294us) server_negotiation.cc:407] Connection header received
0924 10:36:33.854330 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:33.854333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:33.854386 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:33.854483 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:36:33.855363 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.856192 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.857418 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.857771 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.857893 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.859617 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:36:33.859622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:36:33.859625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:36:33.859654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:36:33.861474 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.861991 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.861995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.861997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.862091 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:36:33.862451 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:36:33.862454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:36:33.862456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:36:33.862700 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:36:33.862833 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:33.863112 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 10:36:33.863244 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:36:33.867235 14074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:36:33.871646 14074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:36:33.892197 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.882778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:44975 (local address 127.0.0.1:51386)
0924 10:36:33.885485 (+ 2707us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:33.885546 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:33.885680 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:33.887004 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:33.887012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:33.887036 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:36:33.887170 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.887177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.888729 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.888733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.890228 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.890240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.891604 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.891608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.891728 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.891769 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:36:33.891886 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:36:33.892027 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2510,"thread_start_us":123,"threads_started":1}
I20260924 10:36:33.893431 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.882950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.2.193.129:44975)
0924 10:36:33.886183 (+ 3233us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:33.886189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:33.886214 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:36:33.886292 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:33.886298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:33.886588 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:33.886756 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:36:33.887339 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.888590 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.890377 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.891245 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.891372 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.891534 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:33.893139 (+ 1605us) server_negotiation.cc:299] Negotiation successful
0924 10:36:33.893282 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3130,"thread_start_us":67,"threads_started":1}
I20260924 10:36:33.896891 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.884055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:37881 (local address 127.0.0.1:51570)
0924 10:36:33.889407 (+ 5352us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:33.889424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:33.890415 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:33.891220 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:33.891223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:33.891240 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:36:33.891351 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.891358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.892798 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.892801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.894915 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.894927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.896445 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.896448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.896571 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.896596 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:36:33.896671 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:36:33.896744 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5279,"thread_start_us":115,"threads_started":1}
I20260924 10:36:33.896950 14221 tablet_service.cc:1467] Processing CreateTablet for tablet edf849ae07e94e9cbc1ac0228ed5a0b0 (DEFAULT_TABLE table=test-table [id=331fc6dd80f245e7835620d5b9ebbb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:36:33.897522 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.884497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.2.193.131:37881)
0924 10:36:33.885979 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:33.885996 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:33.890335 (+ 4339us) server_negotiation.cc:407] Connection header received
0924 10:36:33.890604 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:33.890611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:33.890869 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:33.891050 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:36:33.891506 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.892652 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.895089 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.896032 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.896139 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.896291 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:33.897243 (+ 952us) server_negotiation.cc:299] Negotiation successful
0924 10:36:33.897363 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":157,"threads_started":1}
I20260924 10:36:33.898391 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf849ae07e94e9cbc1ac0228ed5a0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:36:33.901505 14492 tablet_service.cc:1467] Processing CreateTablet for tablet edf849ae07e94e9cbc1ac0228ed5a0b0 (DEFAULT_TABLE table=test-table [id=331fc6dd80f245e7835620d5b9ebbb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:36:33.902715 14492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf849ae07e94e9cbc1ac0228ed5a0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:36:33.908424 14576 tablet_bootstrap.cc:492] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: Bootstrap starting.
I20260924 10:36:33.911077 14576 tablet_bootstrap.cc:654] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: Neither blocks nor log segments found. Creating new log.
I20260924 10:36:33.912135 14576 log.cc:824] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: Log is configured to *not* fsync() on all Append() calls
I20260924 10:36:33.914692 14576 tablet_bootstrap.cc:492] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: No bootstrap required, opened a new log
I20260924 10:36:33.914909 14576 ts_tablet_manager.cc:1397] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260924 10:36:33.920540 14576 raft_consensus.cc:348] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:33.921056 14576 raft_consensus.cc:374] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:36:33.921201 14576 raft_consensus.cc:729] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81d4ddbb8244cf5b8c893d38d0c0f15, State: Initialized, Role: FOLLOWER
I20260924 10:36:33.921818 14576 consensus_queue.cc:260] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:33.923180 14576 ts_tablet_manager.cc:1428] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260924 10:36:33.923631 14288 heartbeater.cc:503] Master 127.2.193.190:34691 was elected leader, sending a full tablet report...
I20260924 10:36:33.924186 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.885389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46859 (local address 127.0.0.1:35906)
0924 10:36:33.892916 (+ 7527us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:33.892930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:33.893008 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:33.916893 (+ 23885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:33.916897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:33.916925 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:36:33.917056 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.917065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.921207 (+ 4142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.921212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.922420 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:33.922433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.923687 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.923691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:33.923789 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.923823 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:36:33.923885 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:36:33.923990 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7467,"thread_start_us":93,"threads_started":1}
I20260924 10:36:33.924918 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:33.885833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.2.193.130:46859)
0924 10:36:33.916295 (+ 30462us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:33.916310 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:33.916346 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:36:33.916440 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:33.916447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:33.916701 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:33.916913 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:36:33.919818 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.921023 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.922623 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:33.923532 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:33.924387 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:33.924611 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:33.924672 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 10:36:33.924773 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30323,"thread_start_us":157,"threads_started":1}
I20260924 10:36:33.928427 14350 tablet_service.cc:1467] Processing CreateTablet for tablet edf849ae07e94e9cbc1ac0228ed5a0b0 (DEFAULT_TABLE table=test-table [id=331fc6dd80f245e7835620d5b9ebbb6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:36:33.929670 14350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf849ae07e94e9cbc1ac0228ed5a0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:36:33.933647 14578 tablet_bootstrap.cc:492] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: Bootstrap starting.
I20260924 10:36:33.939308 14580 tablet_bootstrap.cc:492] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: Bootstrap starting.
I20260924 10:36:33.942420 14580 tablet_bootstrap.cc:654] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: Neither blocks nor log segments found. Creating new log.
I20260924 10:36:33.942634 14578 tablet_bootstrap.cc:654] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: Neither blocks nor log segments found. Creating new log.
I20260924 10:36:33.943395 14580 log.cc:824] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: Log is configured to *not* fsync() on all Append() calls
I20260924 10:36:33.943395 14578 log.cc:824] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: Log is configured to *not* fsync() on all Append() calls
I20260924 10:36:33.945842 14580 tablet_bootstrap.cc:492] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: No bootstrap required, opened a new log
I20260924 10:36:33.946086 14580 ts_tablet_manager.cc:1397] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:36:33.950359 14580 raft_consensus.cc:348] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:33.950723 14580 raft_consensus.cc:374] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:36:33.950834 14580 raft_consensus.cc:729] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d60462f5944c6e80e3e396ecbad6ac, State: Initialized, Role: FOLLOWER
I20260924 10:36:33.951133 14578 tablet_bootstrap.cc:492] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: No bootstrap required, opened a new log
I20260924 10:36:33.951395 14578 ts_tablet_manager.cc:1397] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260924 10:36:33.951481 14580 consensus_queue.cc:260] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:33.952579 14580 ts_tablet_manager.cc:1428] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:36:33.952980 14424 heartbeater.cc:503] Master 127.2.193.190:34691 was elected leader, sending a full tablet report...
I20260924 10:36:33.955608 14578 raft_consensus.cc:348] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:33.955924 14578 raft_consensus.cc:374] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:36:33.956022 14578 raft_consensus.cc:729] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bde9b2fcf2477689e0863cebf90803, State: Initialized, Role: FOLLOWER
I20260924 10:36:33.956605 14578 consensus_queue.cc:260] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:33.957372 14558 heartbeater.cc:503] Master 127.2.193.190:34691 was elected leader, sending a full tablet report...
I20260924 10:36:33.957731 14578 ts_tablet_manager.cc:1428] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 10:36:33.962313 14425 tablet.cc:2369] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:36:33 dist-test-slave-8wc7 krb5kdc[13598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246193, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:36:34.041495 14579 raft_consensus.cc:482] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:36:34.041724 14579 raft_consensus.cc:504] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:34.043085 14579 leader_election.cc:288] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54bde9b2fcf2477689e0863cebf90803 (127.2.193.131:37881), 04d60462f5944c6e80e3e396ecbad6ac (127.2.193.130:46859)
I20260924 10:36:34.050982 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:34.044173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:42353 (local address 127.2.193.131:37881)
0924 10:36:34.044330 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:34.044335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:34.046243 (+ 1908us) server_negotiation.cc:407] Connection header received
0924 10:36:34.046453 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:34.046457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:34.046509 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:34.046570 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:36:34.047085 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.047883 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.049652 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.050257 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.050350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:34.050402 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:34.050753 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 10:36:34.050829 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:36:34.051756 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:34.043271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:37881 (local address 127.2.193.129:42353)
0924 10:36:34.046152 (+ 2881us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:34.046167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:34.046325 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:34.046718 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:34.046721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:34.046740 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:36:34.046926 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:34.046936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.048119 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.048122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:34.049490 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:34.049505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.050507 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.050510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:34.050618 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:34.050640 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:36:34.051419 (+ 779us) client_negotiation.cc:240] Negotiation successful
0924 10:36:34.051554 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2802,"thread_start_us":126,"threads_started":1}
I20260924 10:36:34.052619 14512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf849ae07e94e9cbc1ac0228ed5a0b0" candidate_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bde9b2fcf2477689e0863cebf90803" is_pre_election: true
I20260924 10:36:34.052965 14512 raft_consensus.cc:2454] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e81d4ddbb8244cf5b8c893d38d0c0f15 in term 0.
I20260924 10:36:34.053574 14173 leader_election.cc:302] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54bde9b2fcf2477689e0863cebf90803, e81d4ddbb8244cf5b8c893d38d0c0f15; no voters:
I20260924 10:36:34.053965 14579 raft_consensus.cc:2781] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:36:34.054052 14579 raft_consensus.cc:482] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:36:34.054097 14579 raft_consensus.cc:3037] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:36:34.054829 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:34.044310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:32927 (local address 127.2.193.130:46859)
0924 10:36:34.044430 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:34.044434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:34.049411 (+ 4977us) server_negotiation.cc:407] Connection header received
0924 10:36:34.049602 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:34.049606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:34.049655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:34.049703 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:36:34.050104 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.050901 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.053537 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.054163 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.054253 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:34.054300 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:34.054635 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 10:36:34.054713 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:36:34.055233 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:34.043702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46859 (local address 127.2.193.129:32927)
0924 10:36:34.049314 (+ 5612us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:34.049330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:34.049480 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:34.049836 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:36:34.049839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:36:34.049855 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:36:34.049966 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:34.049971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.051085 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.051089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:34.053399 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:36:34.053409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.054392 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.054395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:36:34.054498 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:34.054519 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:36:34.055022 (+ 503us) client_negotiation.cc:240] Negotiation successful
0924 10:36:34.055096 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5532,"thread_start_us":112,"threads_started":1}
I20260924 10:36:34.055446 14579 raft_consensus.cc:504] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:34.055820 14579 leader_election.cc:288] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [CANDIDATE]: Term 1 election: Requested vote from peers 54bde9b2fcf2477689e0863cebf90803 (127.2.193.131:37881), 04d60462f5944c6e80e3e396ecbad6ac (127.2.193.130:46859)
I20260924 10:36:34.055817 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf849ae07e94e9cbc1ac0228ed5a0b0" candidate_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d60462f5944c6e80e3e396ecbad6ac" is_pre_election: true
I20260924 10:36:34.056118 14377 raft_consensus.cc:2454] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e81d4ddbb8244cf5b8c893d38d0c0f15 in term 0.
I20260924 10:36:34.056382 14512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf849ae07e94e9cbc1ac0228ed5a0b0" candidate_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bde9b2fcf2477689e0863cebf90803"
I20260924 10:36:34.056440 14378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf849ae07e94e9cbc1ac0228ed5a0b0" candidate_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d60462f5944c6e80e3e396ecbad6ac"
I20260924 10:36:34.056501 14512 raft_consensus.cc:3037] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:36:34.056540 14378 raft_consensus.cc:3037] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:36:34.057796 14512 raft_consensus.cc:2454] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e81d4ddbb8244cf5b8c893d38d0c0f15 in term 1.
I20260924 10:36:34.057796 14378 raft_consensus.cc:2454] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e81d4ddbb8244cf5b8c893d38d0c0f15 in term 1.
I20260924 10:36:34.058122 14173 leader_election.cc:302] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54bde9b2fcf2477689e0863cebf90803, e81d4ddbb8244cf5b8c893d38d0c0f15; no voters:
I20260924 10:36:34.058722 14579 raft_consensus.cc:2781] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:36:34.058987 14579 raft_consensus.cc:686] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 1 LEADER]: Becoming Leader. State: Replica: e81d4ddbb8244cf5b8c893d38d0c0f15, State: Running, Role: LEADER
I20260924 10:36:34.059376 14579 consensus_queue.cc:237] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } }
I20260924 10:36:34.062814 14074 catalog_manager.cc:5495] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to e81d4ddbb8244cf5b8c893d38d0c0f15 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81d4ddbb8244cf5b8c893d38d0c0f15" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 44975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 } health_report { overall_health: UNKNOWN } } }
W20260924 10:36:34.069904 14559 tablet.cc:2369] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:36:34.093385 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:34.088157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.2.193.129:44975)
0924 10:36:34.088308 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:36:34.088312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:36:34.088328 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:36:34.088423 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:36:34.088426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:36:34.088495 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:36:34.088658 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:36:34.089064 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.089885 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.091149 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:36:34.091495 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:36:34.091599 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:36:34.092039 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:36:34.092659 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:36:34.092794 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:36:34.093145 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 10:36:34.093237 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:36:34.103113 14378 raft_consensus.cc:1261] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 1 FOLLOWER]: Refusing update from remote peer e81d4ddbb8244cf5b8c893d38d0c0f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:36:34.103315 14512 raft_consensus.cc:1261] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 1 FOLLOWER]: Refusing update from remote peer e81d4ddbb8244cf5b8c893d38d0c0f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:36:34.104130 14579 consensus_queue.cc:1035] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54bde9b2fcf2477689e0863cebf90803" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:36:34.104625 14579 consensus_queue.cc:1035] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04d60462f5944c6e80e3e396ecbad6ac" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:36:34.105350 14289 tablet.cc:2369] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:36:34.116016 14592 mvcc.cc:205] Tried to move back new op lower bound from 7332848411035303936 to 7332848410870648832. Current Snapshot: MvccSnapshot[applied={T|T < 7332848411035303936}]
I20260924 10:36:34.119241 14594 mvcc.cc:205] Tried to move back new op lower bound from 7332848411035303936 to 7332848410870648832. Current Snapshot: MvccSnapshot[applied={T|T < 7332848411035303936}]
I20260924 10:36:34.129525 14073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:36:34.129736 14073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:36:34.132747 14073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf3836fa51c4807aa0acb6289abf8f1: Sending DeleteTablet for 3 replicas of tablet edf849ae07e94e9cbc1ac0228ed5a0b0
I20260924 10:36:34.135275 14221 tablet_service.cc:1514] Processing DeleteTablet for tablet edf849ae07e94e9cbc1ac0228ed5a0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:36:34 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:51386
I20260924 10:36:34.135300 14492 tablet_service.cc:1514] Processing DeleteTablet for tablet edf849ae07e94e9cbc1ac0228ed5a0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:36:34 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:51570
I20260924 10:36:34.139451 14350 tablet_service.cc:1514] Processing DeleteTablet for tablet edf849ae07e94e9cbc1ac0228ed5a0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:36:34 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:35906
I20260924 10:36:34.140226 14600 tablet_replica.cc:330] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15: stopping tablet replica
I20260924 10:36:34.140589 14600 raft_consensus.cc:2229] T edf849ae07e94e9cbc1ac0228ed5a0b0 P e81d4ddbb8244cf5b8c893d38d0c0f15 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:36:34.141626 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 14155
I20260924 10:36:34.143344 14603 tablet_replica.cc:330] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: stopping tablet replica
I20260924 10:36:34.143710 14603 raft_consensus.cc:2229] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:36:34.144070 14603 raft_consensus.cc:2258] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:36:34.149567 14601 tablet_replica.cc:330] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803: stopping tablet replica
I20260924 10:36:34.151680 14601 raft_consensus.cc:2229] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:36:34.152957 14603 ts_tablet_manager.cc:1905] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 04d60462f5944c6e80e3e396ecbad6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:36:34.153156 14061 connection.cc:535] server connection from 127.2.193.129:52887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:36:34.153468 14601 raft_consensus.cc:2258] T edf849ae07e94e9cbc1ac0228ed5a0b0 P 54bde9b2fcf2477689e0863cebf90803 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 10:36:34.153569 14060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:44975 (error 108)
I20260924 10:36:34.154875 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 14294
I20260924 10:36:34.162374 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 14429
I20260924 10:36:34.164707 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:34.164340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:44975 (local address 127.0.0.1:51394)
0924 10:36:34.164511 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:34.164585 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:44975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 10:36:34.164719 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:36:34.164389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:46859 (local address 127.0.0.1:35910)
0924 10:36:34.164511 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:34.164586 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 10:36:34.164938 14058 catalog_manager.cc:4584] TS 04d60462f5944c6e80e3e396ecbad6ac (127.2.193.130:46859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edf849ae07e94e9cbc1ac0228ed5a0b0: Network error: Client connection negotiation failed: client connection to 127.2.193.130:46859: connect: Connection refused (error 111)
W20260924 10:36:34.171379 14572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:36:34.170716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:37881 (local address 127.0.0.1:51576)
0924 10:36:34.170903 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 10:36:34.170919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:36:34.171033 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:36:34.171272 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.131:37881: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 10:36:34.172132 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 14042
I20260924 10:36:34.179195 2822 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:36:34.516088 2822 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:36:34.516254 2822 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:36:34.861984 2822 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:36:34.862 UTC [13628] LOG: received smart shutdown request
2026-09-24 10:36:34.865 UTC [13628] LOG: background worker "logical replication launcher" (PID 13637) exited with exit code 1
2026-09-24 10:36:34.866 UTC [13632] LOG: shutting down
2026-09-24 10:36:35.051 UTC [13628] LOG: database system is shut down
2026-09-24T10:36:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46625 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:36:35 dist-test-slave-8wc7 krb5kdc[14613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:36:35 dist-test-slave-8wc7 krb5kdc[14613](info): set up 2 sockets
Sep 24 10:36:35 dist-test-slave-8wc7 krb5kdc[14613](info): commencing operation
krb5kdc: starting...
W20260924 10:36:36.233443 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:36:36 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:36:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/rangeradmin_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/rangeradmin_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/rangerlookup_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/rangerlookup_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
I20260924 10:36:36.408022 2822 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBKECxY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/postgres -l logfile start
2026-09-24 10:36:38.974 UTC [14643] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:36:38.974 UTC [14643] LOG: listening on IPv4 address "127.2.193.148", port 44265
2026-09-24 10:36:38.978 UTC [14643] LOG: listening on Unix socket "/tmp/.s.PGSQL.44265"
2026-09-24 10:36:39.003 UTC [14646] LOG: database system was shut down at 2026-09-24 10:36:37 UTC
2026-09-24 10:36:39.006 UTC [14643] LOG: database system is ready to accept connections
I20260924 10:36:39.957114 2822 mini_postgres.cc:96] Postgres bound to 44265
2026-09-24 10:36:39.963 UTC [14654] FATAL: database "slave" does not exist
127.2.193.148:44265 - accepting connections
I20260924 10:36:39.964031 2822 mini_ranger.cc:161] Starting Ranger...
I20260924 10:36:39.979204 2822 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:36:40.460992 2822 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:36:40.461081 2822 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-admin
2026-09-24 10:36:40,731 [I] DB FLAVOR :POSTGRES
2026-09-24 10:36:40,731 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:36:40,731 [I] Checking connection..
2026-09-24 10:36:40,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:36:40,971 [I] Checking connection passed.
2026-09-24 10:36:40,972 [I] --------- Verifying version history table ---------
2026-09-24 10:36:40,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:36:41,240 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:36:41,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:36:41,475 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:36:41,476 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:36:41,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:36:41.738 UTC [14718] WARNING: there is no transaction in progress
2026-09-24 10:36:41,745 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:36:41,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:36:41,980 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:36:41,980 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:36:41,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:36:42,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:36:42,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8wc7.c.gcp-upstream.internal','N') ;"
2026-09-24 10:36:42,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:36:43,005 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:36:43,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:36:43,268 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:36:43,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:36:43,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:36:43,773 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:36:43,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:36:44.710 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.726 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.742 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.758 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.773 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.793 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.859 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.866 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.880 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.889 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.900 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.913 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.923 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.929 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.935 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:44.942 UTC [14826] WARNING: there is no transaction in progress
2026-09-24 10:36:45,566 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:36:45,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:36:45,795 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:36:45,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:36:46,050 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:36:46,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:36:46,305 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:36:46,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:36:46,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:36:46,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8wc7.c.gcp-upstream.internal';"
2026-09-24 10:36:47,074 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:36:47,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:36:47,322 [I] DB_PATCHES have already been applied
I20260924 10:36:47.328431 2822 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBKECxY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:36:47.328505 2822 mini_ranger.cc:193] Using host: 127.2.193.148
I20260924 10:36:47.331979 2822 mini_ranger.cc:227] Ranger admin URL: http://127.2.193.148:48645
Sep 24, 2026 10:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:36:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:36:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:36:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:36:48.456554 2822 mini_ranger.cc:160] Time spent starting Ranger: real 8.493s user 0.000s sys 0.010s
Sep 24, 2026 10:37:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:37:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 10:37:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:37:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:37:09 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:37:10.604859 2822 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.193.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.193.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/rangerkms_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.193.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/rangerkms_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.193.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.193.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.193.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/HTTP_127.2.193.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:37:10 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246230, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:37:10.698438 2822 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:37:10.708446 2822 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:37:11.657802 2822 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:37:11.657896 2822 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms
I20260924 10:37:11.657917 2822 mini_ranger_kms.cc:227] Using postgres at 127.2.193.148:44265
2026-09-24 10:37:11,834 [I] DB FLAVOR :POSTGRES
2026-09-24 10:37:11,834 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:37:11,834 [I] Checking connection
2026-09-24 10:37:11,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:37:12,092 [I] connection success
2026-09-24 10:37:12,092 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:37:12,092 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:37:12,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:37:12,333 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:37:12,333 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:37:12,333 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:37:12,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.193.148:44265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:37:12,594 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:37:13.097419 2822 mini_ranger_kms.cc:310] Created kms service
I20260924 10:37:13.283164 2822 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:37:13.330900 2822 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:37:13 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246230, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:13.743275 2822 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:37:13.743430 2822 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBKECxY/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBKECxY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:37:13.743451 2822 mini_ranger_kms.cc:240] Using host: 127.2.193.148
I20260924 10:37:13.747684 2822 mini_ranger_kms.cc:276] Ranger KMS PID: 15006
I20260924 10:37:13.747773 2822 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.193.148:40019
Sep 24, 2026 10:37:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:37:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:37:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:37:15.984859 2822 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.286s user 0.004s sys 0.157s
I20260924 10:37:15.984999 2822 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:37:15.985069 2822 mini_ranger_kms.cc:401] 127.2.193.148:40019/kms/v1/keys
Sep 24 10:37:16 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246236, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.193.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:37:16 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246236, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.193.148@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
Sep 24 10:37:16 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246236, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.193.148@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
Sep 24 10:37:19 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:37:19.170140 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34345
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:34345
--ranger_config_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:19.284255 15057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:19.284588 15057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:19.284680 15057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:19.288851 15057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:19.288949 15057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:19.288983 15057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:19.289026 15057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:19.289060 15057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:19.294852 15057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.193.148:40019/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:34345
--ranger_config_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:19.296036 15057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:19.297140 15057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:19.306429 15063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:19.306424 15065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:19.306725 15057 server_base.cc:1034] running on GCE node
W20260924 10:37:19.306910 15062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:19.307540 15057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:19.308712 15057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:19.309900 15057 hybrid_clock.cc:648] HybridClock initialized: now 1790246239309880 us; error 44 us; skew 500 ppm
Sep 24 10:37:19 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246239, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:19.318950 15057 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:19.320096 15057 webserver.cc:460] Webserver started at http://127.2.193.190:45717/ using document root <none> and password file <none>
I20260924 10:37:19.320689 15057 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:19.320747 15057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:19.320981 15057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:37:19 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246239, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:19.369685 15057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "c59fbb1134524e139b41eab5fe297f14"
format_stamp: "Formatted at 2026-09-24 10:37:19 on dist-test-slave-8wc7"
server_key: "b1fba2375ed705659c891b66b8057833"
server_key_iv: "ad6b3b64e13067942af870805865b1a7"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:19.370332 15057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "c59fbb1134524e139b41eab5fe297f14"
format_stamp: "Formatted at 2026-09-24 10:37:19 on dist-test-slave-8wc7"
server_key: "b1fba2375ed705659c891b66b8057833"
server_key_iv: "ad6b3b64e13067942af870805865b1a7"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:19.374347 15057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:37:19.388877 15072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:19.390503 15057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:37:19.390652 15057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "c59fbb1134524e139b41eab5fe297f14"
format_stamp: "Formatted at 2026-09-24 10:37:19 on dist-test-slave-8wc7"
server_key: "b1fba2375ed705659c891b66b8057833"
server_key_iv: "ad6b3b64e13067942af870805865b1a7"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:19.390798 15057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:19.406006 15057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:19.409044 15057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:19.409276 15057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:19.417413 15057 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:34345
I20260924 10:37:19.417423 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:34345 every 8 connection(s)
I20260924 10:37:19.418509 15057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:19.421463 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:19.427114 15125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Bootstrap starting.
I20260924 10:37:19.428350 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15057
I20260924 10:37:19.428495 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:37:19.428778 2822 external_mini_cluster.cc:1428] Setting key 9bd1881d74fd2f4fb6a3314c922f5219
I20260924 10:37:19.430099 15125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:19.431087 15125 log.cc:824] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:19.433432 15125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: No bootstrap required, opened a new log
I20260924 10:37:19.437743 15125 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:19.438011 15125 raft_consensus.cc:374] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:19.438086 15125 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59fbb1134524e139b41eab5fe297f14, State: Initialized, Role: FOLLOWER
I20260924 10:37:19.438565 15125 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:19.438717 15125 raft_consensus.cc:388] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:37:19.438798 15125 raft_consensus.cc:482] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:37:19.438908 15125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:37:19 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:19.439955 15125 raft_consensus.cc:504] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:19.440382 15125 leader_election.cc:302] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c59fbb1134524e139b41eab5fe297f14; no voters:
I20260924 10:37:19.440781 15125 leader_election.cc:288] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:37:19.441144 15130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:19.441426 15130 raft_consensus.cc:686] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 LEADER]: Becoming Leader. State: Replica: c59fbb1134524e139b41eab5fe297f14, State: Running, Role: LEADER
I20260924 10:37:19.441787 15130 consensus_queue.cc:237] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:19.442320 15125 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:19.443060 15130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c59fbb1134524e139b41eab5fe297f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } } }
I20260924 10:37:19.443252 15130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:19.443485 15130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c59fbb1134524e139b41eab5fe297f14. Latest consensus state: current_term: 1 leader_uuid: "c59fbb1134524e139b41eab5fe297f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } } }
I20260924 10:37:19.443605 15130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:19.443894 15138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:19.446384 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:19.430025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.2.193.190:34345)
0924 10:37:19.430547 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:19.430564 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:19.430606 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:37:19.431635 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:19.431656 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:19.432056 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:19.432424 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:19.433347 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:19.434820 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:19.436048 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:19.436810 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:19.436979 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:19.440013 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:19.440019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:19.440039 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:19.440121 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:19.442273 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:19.442856 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:19.442861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:19.442868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:19.442995 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:19.443364 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:19.443368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:19.443369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:19.443925 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:19.444911 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:19.445098 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 10:37:19.445413 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":133,"threads_started":1}
I20260924 10:37:19.448537 15138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:19.449391 15125 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:37:19.458361 15138 catalog_manager.cc:1347] Generated new cluster ID: 2c8fcc0cea5142ea92ee290624d966e3
I20260924 10:37:19.458501 15138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:37:19.473323 15138 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:37:19.474783 15138 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:37:19.481645 15138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Generated new TSK 0
I20260924 10:37:19.482626 15138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:37:20.228837 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34345
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:20.366683 15170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:20.367012 15170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:20.367127 15170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:20.371469 15170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:20.371578 15170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:20.371687 15170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:37:20.377537 15170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.193.148:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:20.378775 15170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:20.379904 15170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:20.390919 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:20.391089 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:20.390942 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:20.391407 15170 server_base.cc:1034] running on GCE node
I20260924 10:37:20.392045 15170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:20.392896 15170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:20.396274 15170 hybrid_clock.cc:648] HybridClock initialized: now 1790246240396226 us; error 57 us; skew 500 ppm
Sep 24 10:37:20 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:20.400120 15170 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:37:20.401379 15170 webserver.cc:460] Webserver started at http://127.2.193.129:42187/ using document root <none> and password file <none>
I20260924 10:37:20.402145 15170 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:20.402244 15170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:20.402485 15170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:37:20 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:20.425449 15170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "f42c3cb88c0a44168ff8093d346ad162"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "af63c8333fef51975df895f449414fa5"
server_key_iv: "386c444c88d0b00af123c5e88e7c3d8e"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:20.433975 15170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "f42c3cb88c0a44168ff8093d346ad162"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "af63c8333fef51975df895f449414fa5"
server_key_iv: "386c444c88d0b00af123c5e88e7c3d8e"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:20.439895 15170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:37:20.460235 15189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:20.462085 15170 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260924 10:37:20.462221 15170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "f42c3cb88c0a44168ff8093d346ad162"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "af63c8333fef51975df895f449414fa5"
server_key_iv: "386c444c88d0b00af123c5e88e7c3d8e"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:20.462354 15170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:20.478317 15170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:20.482265 15170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:20.482545 15170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:20.483444 15170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:20.484491 15170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:20.484570 15170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:20.484668 15170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:20.484704 15170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:20.505119 15170 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:45359
I20260924 10:37:20.507083 15170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:37:20.509971 15302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:45359 every 8 connection(s)
I20260924 10:37:20.516184 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15170
I20260924 10:37:20.516357 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:37:20.516678 2822 external_mini_cluster.cc:1428] Setting key 8549e21915c57bbd77d2bfde636b658f
Sep 24 10:37:20 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:20.531533 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:20.512997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34345 (local address 127.2.193.129:55079)
0924 10:37:20.514308 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:20.514369 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:20.515419 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:20.516611 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:20.516622 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:20.517064 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:20.519016 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:20.519042 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.520260 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.520264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:20.521771 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:20.521784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.522580 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.522583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:20.522791 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:20.523541 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:20.523562 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:20.525362 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:20.527828 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:20.527834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:20.527853 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:20.528242 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:20.528588 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:20.528591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:20.528593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:20.528802 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:20.529403 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:20.529411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:20.529794 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 10:37:20.530033 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 10:37:20.530415 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":190,"threads_started":1}
I20260924 10:37:20.532361 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:20.512466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:55079 (local address 127.2.193.190:34345)
0924 10:37:20.515874 (+ 3408us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:20.515880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:20.515896 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:37:20.515958 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:20.515962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:20.516020 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:20.516116 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:20.519208 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.520105 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.521922 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.522294 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.522390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:20.525596 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:20.525601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:20.525604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:20.525634 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:20.527651 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:20.528370 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:20.528373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:20.528375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:20.528477 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:20.528930 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:20.528934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:20.528937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:20.529245 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:20.529343 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:20.532046 (+ 2703us) server_negotiation.cc:299] Negotiation successful
0924 10:37:20.532191 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3321,"thread_start_us":70,"threads_started":1}
I20260924 10:37:20.536353 15303 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34345
I20260924 10:37:20.536643 15303 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:20.537472 15303 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:20.539461 15089 ts_manager.cc:194] Registered new tserver with Master: f42c3cb88c0a44168ff8093d346ad162 (127.2.193.129:45359)
I20260924 10:37:20.541090 15089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:55079
Sep 24 10:37:20 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:37:20.571959 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34345
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:20.719393 15309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:20.719712 15309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:20.719834 15309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:20.725972 15309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:20.726086 15309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:20.726197 15309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:37:20.732074 15309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.193.148:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:20.733359 15309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:20.734655 15309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:20.744921 15315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:20.744942 15318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:20.745513 15309 server_base.cc:1034] running on GCE node
W20260924 10:37:20.745569 15316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:20.746014 15309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:20.746737 15309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:20.747915 15309 hybrid_clock.cc:648] HybridClock initialized: now 1790246240747885 us; error 47 us; skew 500 ppm
Sep 24 10:37:20 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:20.750891 15309 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:37:20.752039 15309 webserver.cc:460] Webserver started at http://127.2.193.130:41963/ using document root <none> and password file <none>
I20260924 10:37:20.752652 15309 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:20.752748 15309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:20.752992 15309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:37:20 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:20.768767 15309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "76df57f032824042879d9465e1c8f284"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "2e4df40c31d1d10217bbed14f1ec48c3"
server_key_iv: "6cb1f44d91b0ee8e52abee70203605bb"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:20.770205 15309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "76df57f032824042879d9465e1c8f284"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "2e4df40c31d1d10217bbed14f1ec48c3"
server_key_iv: "6cb1f44d91b0ee8e52abee70203605bb"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:20.781441 15309 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.000s
I20260924 10:37:20.808763 15325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:20.810724 15309 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260924 10:37:20.810873 15309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "76df57f032824042879d9465e1c8f284"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "2e4df40c31d1d10217bbed14f1ec48c3"
server_key_iv: "6cb1f44d91b0ee8e52abee70203605bb"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:20.811015 15309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:20.833576 15309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:20.837019 15309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:20.837268 15309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:20.837996 15309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:20.838979 15309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:20.839051 15309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:20.839156 15309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:20.839203 15309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:20.850430 15309 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:34047
I20260924 10:37:20.851702 15309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:37:20.854555 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15309
I20260924 10:37:20.854681 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:37:20.854986 2822 external_mini_cluster.cc:1428] Setting key 0467de261bfbfb283d91c73edbc662e9
I20260924 10:37:20.852556 15438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:34047 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Sep 24 10:37:20 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:20.889662 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:20.854885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34345 (local address 127.2.193.130:34051)
0924 10:37:20.861764 (+ 6879us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:20.861839 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:20.863009 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:20.863806 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:20.863816 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:20.864213 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:20.864939 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:20.864960 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.866219 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.866223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:20.867700 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:20.867712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.868504 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.868508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:20.868716 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:20.881400 (+ 12684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:20.881428 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:20.883355 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:20.885852 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:20.885861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:20.885884 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:20.886301 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:20.886663 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:20.886666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:20.886669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:20.886889 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:20.887475 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:20.887482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:20.887857 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 10:37:20.888094 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 10:37:20.888477 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6581,"thread_start_us":148,"threads_started":1}
I20260924 10:37:20.890386 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:20.856053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:34051 (local address 127.2.193.190:34345)
0924 10:37:20.856216 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:20.856221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:20.861979 (+ 5758us) server_negotiation.cc:407] Connection header received
0924 10:37:20.863296 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:20.863300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:20.863362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:20.863459 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:20.865130 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.866041 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.867858 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:20.868216 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:20.868288 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:20.883700 (+ 15412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:20.883705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:20.883708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:20.883743 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:20.885670 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:20.886455 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:20.886458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:20.886460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:20.886550 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:20.887014 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:20.887017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:20.887019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:20.887279 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:20.887413 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:20.890077 (+ 2664us) server_negotiation.cc:299] Negotiation successful
0924 10:37:20.890221 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:37:20.892164 15439 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34345
I20260924 10:37:20.892438 15439 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:20.893041 15439 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:20.894367 15089 ts_manager.cc:194] Registered new tserver with Master: 76df57f032824042879d9465e1c8f284 (127.2.193.130:34047)
I20260924 10:37:20.895109 15089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:34051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:37:20.913658 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34345
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246240, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
W20260924 10:37:21.131932 15444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:21.132316 15444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:21.132378 15444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:21.138938 15444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:21.139034 15444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:21.139119 15444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:37:21.146942 15444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.193.148:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:21.148612 15444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:21.150190 15444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:21.163877 15449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:21.164359 15452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:21.164921 15450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:21.165477 15444 server_base.cc:1034] running on GCE node
I20260924 10:37:21.165972 15444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:21.166761 15444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:21.167929 15444 hybrid_clock.cc:648] HybridClock initialized: now 1790246241167856 us; error 86 us; skew 500 ppm
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:21.179440 15444 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:37:21.180737 15444 webserver.cc:460] Webserver started at http://127.2.193.131:33325/ using document root <none> and password file <none>
I20260924 10:37:21.181572 15444 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:21.181716 15444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:21.182006 15444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:21.201304 15444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "e6988e4e46df4bc4978729e460d86cc3"
format_stamp: "Formatted at 2026-09-24 10:37:21 on dist-test-slave-8wc7"
server_key: "21165878504608efe81ff95232aef6db"
server_key_iv: "36972196d5011fab0b76cd6088c30e07"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:21.202737 15444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "e6988e4e46df4bc4978729e460d86cc3"
format_stamp: "Formatted at 2026-09-24 10:37:21 on dist-test-slave-8wc7"
server_key: "21165878504608efe81ff95232aef6db"
server_key_iv: "36972196d5011fab0b76cd6088c30e07"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:21.215950 15444 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260924 10:37:21.241019 15459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:21.242853 15444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:37:21.243237 15444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "e6988e4e46df4bc4978729e460d86cc3"
format_stamp: "Formatted at 2026-09-24 10:37:21 on dist-test-slave-8wc7"
server_key: "21165878504608efe81ff95232aef6db"
server_key_iv: "36972196d5011fab0b76cd6088c30e07"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:21.243597 15444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:21.265018 15444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:21.268989 15444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:21.269328 15444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:21.270113 15444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:21.271282 15444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:21.271384 15444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:21.271476 15444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:21.271544 15444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:21.282537 15444 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:33537
I20260924 10:37:21.285058 15444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:37:21.291512 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15444
I20260924 10:37:21.291662 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:37:21.291960 2822 external_mini_cluster.cc:1428] Setting key 0b3c72527a6c22c5c235d3781884dcf1
I20260924 10:37:21.287797 15572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:33537 every 8 connection(s)
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:21.311435 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:21.289720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:35399 (local address 127.2.193.190:34345)
0924 10:37:21.289852 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:21.289857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:21.295649 (+ 5792us) server_negotiation.cc:407] Connection header received
0924 10:37:21.296900 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:21.296903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:21.296957 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:21.297007 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:21.298944 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.299788 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.301520 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.301869 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.301949 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:21.306948 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:21.306952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:21.306954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:21.306984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:21.308805 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:21.309523 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:21.309526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:21.309528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:21.309627 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:21.310052 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:21.310055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:21.310057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:21.310291 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:21.310370 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:21.311124 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 10:37:21.311269 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:37:21.313450 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:21.290222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34345 (local address 127.2.193.131:35399)
0924 10:37:21.295466 (+ 5244us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:21.295537 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:21.296637 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:21.297490 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:21.297500 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:21.297873 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:21.298743 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:21.298767 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.299965 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.299968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:21.301374 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:21.301385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.302170 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.302173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:21.302370 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:21.303052 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:21.303074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:21.306744 (+ 3670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:21.308964 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:21.308971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:21.308991 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:21.309404 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:21.309733 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:21.309736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:21.309737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:21.309944 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:21.310442 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:21.310450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:21.310825 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 10:37:21.311938 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0924 10:37:21.312321 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4928,"thread_start_us":156,"threads_started":1}
I20260924 10:37:21.315086 15573 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34345
I20260924 10:37:21.315341 15573 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:21.315923 15573 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:21.318614 15089 ts_manager.cc:194] Registered new tserver with Master: e6988e4e46df4bc4978729e460d86cc3 (127.2.193.131:33537)
I20260924 10:37:21.319327 15089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:35399
I20260924 10:37:21.329891 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:37:21.330646 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15170
I20260924 10:37:21.338076 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15309
I20260924 10:37:21.344596 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15444
I20260924 10:37:21.352299 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15057
I20260924 10:37:21.359268 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34345
--webserver_interface=127.2.193.190
--webserver_port=45717
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:34345
--ranger_config_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:21.474906 15576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:21.475204 15576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:21.475246 15576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:21.479354 15576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:21.479421 15576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:21.479441 15576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:21.479457 15576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:21.479472 15576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:21.485086 15576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:34345
--ranger_config_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=45717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:21.486253 15576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:21.487255 15576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:21.496129 15584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:21.496249 15576 server_base.cc:1034] running on GCE node
W20260924 10:37:21.496132 15581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:21.496191 15582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:21.497006 15576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:21.498227 15576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:21.499420 15576 hybrid_clock.cc:648] HybridClock initialized: now 1790246241499391 us; error 43 us; skew 500 ppm
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:21.502252 15576 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:21.503177 15576 webserver.cc:460] Webserver started at http://127.2.193.190:45717/ using document root <none> and password file <none>
I20260924 10:37:21.503764 15576 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:21.503865 15576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:21.565524 15576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:37:21.568147 15592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:21.569661 15576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:37:21.569821 15576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "c59fbb1134524e139b41eab5fe297f14"
format_stamp: "Formatted at 2026-09-24 10:37:19 on dist-test-slave-8wc7"
server_key: "b1fba2375ed705659c891b66b8057833"
server_key_iv: "ad6b3b64e13067942af870805865b1a7"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:21.570433 15576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:21.603228 15576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:21.606199 15576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:21.606431 15576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:21.614112 15576 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:34345
I20260924 10:37:21.614120 15644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:34345 every 8 connection(s)
I20260924 10:37:21.615260 15576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:21.618167 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15576
I20260924 10:37:21.618659 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:45359
--local_ip_for_outbound_sockets=127.2.193.129
--tserver_master_addrs=127.2.193.190:34345
--webserver_port=42187
--webserver_interface=127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:37:21.619920 15645 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:37:21.621759 15645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Bootstrap starting.
I20260924 10:37:21.638639 15645 log.cc:824] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:21.645272 15645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:37:21.645681 15645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Bootstrap complete.
I20260924 10:37:21.649916 15645 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:21.650352 15645 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59fbb1134524e139b41eab5fe297f14, State: Initialized, Role: FOLLOWER
I20260924 10:37:21.650839 15645 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:21.650998 15645 raft_consensus.cc:388] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:37:21.651077 15645 raft_consensus.cc:482] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:37:21.651197 15645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:37:21.653225 15645 raft_consensus.cc:504] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:21.653579 15645 leader_election.cc:302] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c59fbb1134524e139b41eab5fe297f14; no voters:
I20260924 10:37:21.653919 15645 leader_election.cc:288] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:37:21.654017 15649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:37:21.654222 15649 raft_consensus.cc:686] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [term 2 LEADER]: Becoming Leader. State: Replica: c59fbb1134524e139b41eab5fe297f14, State: Running, Role: LEADER
I20260924 10:37:21.654525 15649 consensus_queue.cc:237] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } }
I20260924 10:37:21.655176 15645 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:21.655441 15649 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c59fbb1134524e139b41eab5fe297f14. Latest consensus state: current_term: 2 leader_uuid: "c59fbb1134524e139b41eab5fe297f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } } }
I20260924 10:37:21.655570 15649 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:21.655921 15653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:21.656060 15650 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c59fbb1134524e139b41eab5fe297f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59fbb1134524e139b41eab5fe297f14" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34345 } } }
I20260924 10:37:21.656234 15650 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:21.659307 15653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:21.659850 15645 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:37:21.660017 15653 catalog_manager.cc:1259] Loaded cluster ID: 2c8fcc0cea5142ea92ee290624d966e3
I20260924 10:37:21.660063 15653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:37:21.661955 15653 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:37:21.662549 15653 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Loaded TSK: 0
I20260924 10:37:21.663586 15653 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:37:21.764004 15647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:21.764375 15647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:21.764451 15647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:21.770643 15647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:21.770735 15647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:21.770831 15647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:37:21.778832 15647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:45359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=42187
--tserver_master_addrs=127.2.193.190:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:21.780162 15647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:21.781483 15647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:21.793865 15685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:21.794140 15683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:21.794341 15647 server_base.cc:1034] running on GCE node
W20260924 10:37:21.794579 15682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:21.795104 15647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:21.796000 15647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:21.797216 15647 hybrid_clock.cc:648] HybridClock initialized: now 1790246241797188 us; error 44 us; skew 500 ppm
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:21.801091 15647 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:37:21.802320 15647 webserver.cc:460] Webserver started at http://127.2.193.129:42187/ using document root <none> and password file <none>
I20260924 10:37:21.803066 15647 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:21.803185 15647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:21.849670 15647 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:37:21.853040 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:21.854580 15647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:37:21.854743 15647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "f42c3cb88c0a44168ff8093d346ad162"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "af63c8333fef51975df895f449414fa5"
server_key_iv: "386c444c88d0b00af123c5e88e7c3d8e"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:21.855577 15647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:21.879586 15647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:21.883473 15647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:21.883683 15647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:21.884387 15647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:21.885492 15647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:21.885564 15647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:21.885619 15647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:21.885648 15647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:21.898209 15647 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:45359
I20260924 10:37:21.899638 15647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:37:21.901796 15806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:45359 every 8 connection(s)
I20260924 10:37:21.907799 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15647
I20260924 10:37:21.908316 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:34047
--local_ip_for_outbound_sockets=127.2.193.130
--tserver_master_addrs=127.2.193.190:34345
--webserver_port=41963
--webserver_interface=127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:37:21 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246241, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:21.923105 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:21.904331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34345 (local address 127.2.193.129:56471)
0924 10:37:21.906989 (+ 2658us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:21.907048 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:21.908215 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:21.910409 (+ 2194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:21.910415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:21.910748 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:21.911334 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:21.911350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.913149 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.913154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:21.914343 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:21.914354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.915173 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.915178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:21.915358 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:21.916025 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:21.916042 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:21.917501 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:21.919706 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:21.919713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:21.919728 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:21.919998 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:21.920365 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:21.920369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:21.920371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:21.920527 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:21.921151 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:21.921158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:21.921454 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 10:37:21.921657 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 10:37:21.922033 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2411,"thread_start_us":163,"threads_started":1}
I20260924 10:37:21.923092 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:21.905929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:56471 (local address 127.2.193.190:34345)
0924 10:37:21.908767 (+ 2838us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:21.908795 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:21.908843 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:37:21.909157 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:21.909199 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:21.909622 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:21.910005 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:21.911527 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.912973 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.914481 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:21.914996 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:21.915186 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:21.917723 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:21.917728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:21.917742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:21.917771 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:21.919561 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:21.920142 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:21.920147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:21.920152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:21.920266 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:21.920634 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:21.920642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:21.920643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:21.921034 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:21.921181 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:21.921796 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 10:37:21.922094 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2647,"thread_start_us":121,"threads_started":1}
I20260924 10:37:21.926790 15807 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34345
I20260924 10:37:21.927079 15807 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:21.927728 15807 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
W20260924 10:37:21.930094 15807 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:34345 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:37:21.930253 15807 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:21.930441 15807 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:21.931325 15807 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:21.931519 15807 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
W20260924 10:37:21.932348 15807 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:37:22.079424 15811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:22.079867 15811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:22.079941 15811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:22.086565 15811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:22.086653 15811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:22.086755 15811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:37:22.095405 15811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:34047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=41963
--tserver_master_addrs=127.2.193.190:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:22.096906 15811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:22.098429 15811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:22.111163 15819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:22.111570 15816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:22.111603 15817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:22.111967 15811 server_base.cc:1034] running on GCE node
I20260924 10:37:22.112485 15811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:22.113335 15811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:22.114537 15811 hybrid_clock.cc:648] HybridClock initialized: now 1790246242114512 us; error 41 us; skew 500 ppm
Sep 24 10:37:22 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:22.118258 15811 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:37:22.119375 15811 webserver.cc:460] Webserver started at http://127.2.193.130:41963/ using document root <none> and password file <none>
I20260924 10:37:22.120136 15811 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:22.120254 15811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:37:22 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:22.139884 15811 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:37:22.143185 15827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:22.144901 15811 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 10:37:22.145082 15811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "76df57f032824042879d9465e1c8f284"
format_stamp: "Formatted at 2026-09-24 10:37:20 on dist-test-slave-8wc7"
server_key: "2e4df40c31d1d10217bbed14f1ec48c3"
server_key_iv: "6cb1f44d91b0ee8e52abee70203605bb"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:22.145766 15811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:22.179170 15811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:22.183269 15811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:22.183496 15811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:22.184252 15811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:22.185348 15811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:22.185415 15811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:22.185470 15811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:22.185498 15811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:22.196936 15811 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:34047
I20260924 10:37:22.197005 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:34047 every 8 connection(s)
I20260924 10:37:22.198863 15811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:37:22.205364 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15811
I20260924 10:37:22.205929 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.193.148:40019/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:33537
--local_ip_for_outbound_sockets=127.2.193.131
--tserver_master_addrs=127.2.193.190:34345
--webserver_port=33325
--webserver_interface=127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:37:22 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:22.221412 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:22.202767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:56771 (local address 127.2.193.190:34345)
0924 10:37:22.202926 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:22.202931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:22.205626 (+ 2695us) server_negotiation.cc:407] Connection header received
0924 10:37:22.207236 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:22.207239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:22.207293 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:22.207467 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:22.209148 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.210059 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.211943 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.212320 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.212421 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:22.216002 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:22.216009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:22.216015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:22.216047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:22.218018 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:22.219246 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:22.219249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:22.219251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:22.219365 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:22.219886 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:22.219891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:22.219893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:22.220148 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:22.220269 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:22.221037 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 10:37:22.221229 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:37:22.223196 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:22.201677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34345 (local address 127.2.193.130:56771)
0924 10:37:22.205400 (+ 3723us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:22.205480 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:22.206931 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:22.207831 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:22.207842 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:22.208230 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:22.208968 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:22.208985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.210229 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.210233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:22.211757 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:22.211773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.212676 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.212681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:22.212894 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:22.213789 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:22.213812 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:22.215747 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:22.218173 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:22.218184 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:22.218208 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:22.219102 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:22.219488 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:22.219494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:22.219498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:22.219743 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:22.220286 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:22.220295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:22.220720 (+ 425us) client_negotiation.cc:769] Sending connection context
0924 10:37:22.221677 (+ 957us) client_negotiation.cc:240] Negotiation successful
0924 10:37:22.222073 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3399,"thread_start_us":161,"threads_started":1}
I20260924 10:37:22.224607 15942 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34345
I20260924 10:37:22.224890 15942 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:22.225539 15942 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
W20260924 10:37:22.227418 15942 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:34345 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:37:22.227574 15942 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:22.227778 15942 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:22.228739 15942 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:22.228929 15942 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
W20260924 10:37:22.229755 15942 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:37:22.351217 15945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:22.351641 15945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:22.351760 15945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:22.356707 15945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:22.356796 15945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:22.356887 15945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:37:22.363821 15945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.193.148:40019/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:33537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=33325
--tserver_master_addrs=127.2.193.190:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:22.365010 15945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:22.366437 15945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:22.377429 15950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:22.377676 15953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:22.377449 15951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:22.378237 15945 server_base.cc:1034] running on GCE node
I20260924 10:37:22.378741 15945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:22.379503 15945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:22.380662 15945 hybrid_clock.cc:648] HybridClock initialized: now 1790246242380629 us; error 54 us; skew 500 ppm
Sep 24 10:37:22 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:22.384299 15945 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:37:22.385447 15945 webserver.cc:460] Webserver started at http://127.2.193.131:33325/ using document root <none> and password file <none>
I20260924 10:37:22.386153 15945 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:22.386241 15945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:37:22 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:22.415316 15945 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 10:37:22.418928 15961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:22.421613 15945 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:37:22.421793 15945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "e6988e4e46df4bc4978729e460d86cc3"
format_stamp: "Formatted at 2026-09-24 10:37:21 on dist-test-slave-8wc7"
server_key: "21165878504608efe81ff95232aef6db"
server_key_iv: "36972196d5011fab0b76cd6088c30e07"
server_key_version: "kuduclusterkey@0"
I20260924 10:37:22.422598 15945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:22.463174 15945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:22.466814 15945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:22.467028 15945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:22.467782 15945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:22.468847 15945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:22.468912 15945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:22.468968 15945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:22.469005 15945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:22.480221 15945 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:33537
I20260924 10:37:22.480312 16081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:33537 every 8 connection(s)
I20260924 10:37:22.481617 15945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:37:22.487814 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 15945
Sep 24 10:37:22 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:22.497952 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:22.484147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34345 (local address 127.2.193.131:38521)
0924 10:37:22.484699 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:22.484743 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:22.485660 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:22.486211 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:22.486219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:22.486504 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:22.486988 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:22.487003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.487844 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.487848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:22.489054 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:22.489064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.489716 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.489720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:22.489933 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:22.490594 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:22.490616 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:22.492373 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:22.494589 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:22.494596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:22.494615 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:22.494959 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:22.495273 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:22.495277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:22.495279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:22.495462 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:22.495948 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:22.495955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:22.496278 (+ 323us) client_negotiation.cc:769] Sending connection context
0924 10:37:22.496503 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 10:37:22.496841 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":152,"threads_started":1}
I20260924 10:37:22.498646 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:22.483597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:38521 (local address 127.2.193.190:34345)
0924 10:37:22.483745 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:22.483749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:22.484864 (+ 1115us) server_negotiation.cc:407] Connection header received
0924 10:37:22.485876 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:22.485880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:22.485947 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:22.486031 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:22.487138 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.487714 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.489198 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.489567 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.489667 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:22.492599 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:22.492604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:22.492607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:22.492636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:22.494441 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:22.495081 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:22.495084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:22.495086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:22.495173 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:22.495588 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:22.495591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:22.495593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:22.495808 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:22.495896 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:22.498347 (+ 2451us) server_negotiation.cc:299] Negotiation successful
0924 10:37:22.498491 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:37:22.500324 16082 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34345
I20260924 10:37:22.500579 16082 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:22.501133 16082 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:22.502733 15606 ts_manager.cc:194] Registered new tserver with Master: e6988e4e46df4bc4978729e460d86cc3 (127.2.193.131:33537)
I20260924 10:37:22.504249 15606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:38521
I20260924 10:37:22.504729 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:22.488584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.2.193.190:34345)
0924 10:37:22.488879 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:22.488886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:22.489484 (+ 598us) server_negotiation.cc:407] Connection header received
0924 10:37:22.489594 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:22.489597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:22.489649 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:22.489747 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:22.491024 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.492611 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.498556 (+ 5945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:22.498926 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:22.499022 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:22.500824 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:22.500828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:22.500831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:22.500856 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:22.502690 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:22.503262 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:22.503265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:22.503267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:22.503361 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:22.503692 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:22.503696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:22.503697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:22.503903 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:22.504090 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:22.504271 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 10:37:22.504404 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":120,"threads_started":1}
Sep 24 10:37:22 dist-test-slave-8wc7 krb5kdc[14613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:22.932612 15807 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:22.932929 15807 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:22.933761 15606 ts_manager.cc:194] Registered new tserver with Master: f42c3cb88c0a44168ff8093d346ad162 (127.2.193.129:45359)
I20260924 10:37:22.934512 15606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:56471
Sep 24 10:37:23 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
I20260924 10:37:23.230019 15942 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:23.230518 15942 heartbeater.cc:511] Master 127.2.193.190:34345 requested a full tablet report, sending...
I20260924 10:37:23.231585 15606 ts_manager.cc:194] Registered new tserver with Master: 76df57f032824042879d9465e1c8f284 (127.2.193.130:34047)
I20260924 10:37:23.232370 15606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:56771
I20260924 10:37:23.238056 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:37:23.249411 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.239654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.2.193.190:34345)
0924 10:37:23.239912 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:23.239916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:23.240211 (+ 295us) server_negotiation.cc:407] Connection header received
0924 10:37:23.240339 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:23.240342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:23.240393 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:23.240490 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:23.241370 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.242206 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.243395 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.243751 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.243854 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.245625 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:23.245629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:23.245632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:23.245662 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:23.247493 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:23.248006 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:23.248010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:23.248012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:23.248105 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:23.248429 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:23.248432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:23.248434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:23.248656 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:23.248785 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:23.249057 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 10:37:23.249214 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260924 10:37:23.254159 15606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:37:23.258882 15606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:37:23.281129 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.271984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:45359 (local address 127.0.0.1:51094)
0924 10:37:23.273667 (+ 1683us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:23.273679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:23.273752 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:23.274528 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:23.274535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:23.274562 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:23.274717 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.274723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.277144 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.277148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.278602 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.278613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.279809 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.279813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.279940 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.279998 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 10:37:23.280084 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:37:23.280967 (+ 883us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":91,"threads_started":1}
I20260924 10:37:23.281985 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.272369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.2.193.129:45359)
0924 10:37:23.273986 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:23.273993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:23.274018 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:37:23.274096 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:23.274102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:23.274337 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:23.274504 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:23.274902 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.276262 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.278760 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.279670 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.281522 (+ 1852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.281683 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:23.281743 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:37:23.281839 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":98,"threads_started":1}
I20260924 10:37:23.283874 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.272196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:33537 (local address 127.0.0.1:43338)
0924 10:37:23.277205 (+ 5009us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:23.277218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:23.277304 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:23.277963 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:23.277966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:23.277981 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:23.278066 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.278071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.279527 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.279530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.280654 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.280665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.283469 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.283473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.283570 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.283594 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:37:23.283670 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:37:23.283736 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4922,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260924 10:37:23.284309 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.272613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.2.193.131:33537)
0924 10:37:23.272959 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:23.272966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:23.277402 (+ 4436us) server_negotiation.cc:407] Connection header received
0924 10:37:23.277488 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:23.277493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:23.277677 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:23.277818 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:23.278186 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.279399 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.282520 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.283327 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.283442 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.283578 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:23.284072 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 10:37:23.284177 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260924 10:37:23.285264 15741 tablet_service.cc:1467] Processing CreateTablet for tablet 33080143eaa7468cb9f3cca32551caa8 (DEFAULT_TABLE table=test-table [id=5960c392637149a2b663c0d2b2c1e11f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:23.286451 15741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33080143eaa7468cb9f3cca32551caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:23.287204 16016 tablet_service.cc:1467] Processing CreateTablet for tablet 33080143eaa7468cb9f3cca32551caa8 (DEFAULT_TABLE table=test-table [id=5960c392637149a2b663c0d2b2c1e11f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:23.288285 16016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33080143eaa7468cb9f3cca32551caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:23.290480 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.270776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34047 (local address 127.0.0.1:43958)
0924 10:37:23.273209 (+ 2433us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:23.273268 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:23.273400 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:23.275383 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:23.275386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:23.275404 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:23.275495 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.275500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.280808 (+ 5308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.280811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.281976 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.281987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.289998 (+ 8011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.290003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.290134 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.290163 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:37:23.290226 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:37:23.290313 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2232,"thread_start_us":125,"threads_started":1}
I20260924 10:37:23.292317 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.270886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.2.193.130:34047)
0924 10:37:23.274894 (+ 4008us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:23.274902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:23.274938 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:37:23.275015 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:23.275020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:23.275229 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:23.275643 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:23.275781 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.276972 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.283683 (+ 6711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.289803 (+ 6120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.291831 (+ 2028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.292011 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:23.292069 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 10:37:23.292168 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3898,"thread_start_us":109,"threads_started":1}
I20260924 10:37:23.295579 15857 tablet_service.cc:1467] Processing CreateTablet for tablet 33080143eaa7468cb9f3cca32551caa8 (DEFAULT_TABLE table=test-table [id=5960c392637149a2b663c0d2b2c1e11f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:23.296841 15857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33080143eaa7468cb9f3cca32551caa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:23.297338 16109 tablet_bootstrap.cc:492] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Bootstrap starting.
I20260924 10:37:23.297338 16108 tablet_bootstrap.cc:492] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162: Bootstrap starting.
I20260924 10:37:23.299660 16109 tablet_bootstrap.cc:654] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:23.299661 16108 tablet_bootstrap.cc:654] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:23.300535 16109 log.cc:824] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:23.300535 16108 log.cc:824] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:23.302699 16109 tablet_bootstrap.cc:492] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: No bootstrap required, opened a new log
I20260924 10:37:23.302699 16108 tablet_bootstrap.cc:492] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162: No bootstrap required, opened a new log
I20260924 10:37:23.302901 16108 ts_tablet_manager.cc:1397] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:37:23.302901 16109 ts_tablet_manager.cc:1397] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:37:23.307081 16108 raft_consensus.cc:348] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.307379 16108 raft_consensus.cc:374] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:23.307441 16108 raft_consensus.cc:729] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42c3cb88c0a44168ff8093d346ad162, State: Initialized, Role: FOLLOWER
I20260924 10:37:23.307534 16109 raft_consensus.cc:348] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.307760 16109 raft_consensus.cc:374] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:23.307802 16109 raft_consensus.cc:729] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6988e4e46df4bc4978729e460d86cc3, State: Initialized, Role: FOLLOWER
I20260924 10:37:23.307910 16108 consensus_queue.cc:260] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.308945 16108 ts_tablet_manager.cc:1428] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:37:23.309356 15807 heartbeater.cc:503] Master 127.2.193.190:34345 was elected leader, sending a full tablet report...
I20260924 10:37:23.309625 16112 tablet_bootstrap.cc:492] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: Bootstrap starting.
I20260924 10:37:23.312112 16112 tablet_bootstrap.cc:654] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:23.313283 16112 log.cc:824] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:23.314493 16109 consensus_queue.cc:260] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.315590 16112 tablet_bootstrap.cc:492] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: No bootstrap required, opened a new log
I20260924 10:37:23.315608 16109 ts_tablet_manager.cc:1428] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260924 10:37:23.315790 16112 ts_tablet_manager.cc:1397] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:37:23.315965 16082 heartbeater.cc:503] Master 127.2.193.190:34345 was elected leader, sending a full tablet report...
I20260924 10:37:23.322085 16112 raft_consensus.cc:348] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.322417 16112 raft_consensus.cc:374] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:23.322536 16112 raft_consensus.cc:729] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76df57f032824042879d9465e1c8f284, State: Initialized, Role: FOLLOWER
I20260924 10:37:23.323077 16112 consensus_queue.cc:260] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.324270 16112 ts_tablet_manager.cc:1428] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:37:23.324666 15942 heartbeater.cc:503] Master 127.2.193.190:34345 was elected leader, sending a full tablet report...
W20260924 10:37:23.408169 15808 tablet.cc:2369] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:37:23.450976 15943 tablet.cc:2369] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:37:23 dist-test-slave-8wc7 krb5kdc[14613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246242, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for HTTP/127.2.193.148@KRBTEST.COM
W20260924 10:37:23.484512 16083 tablet.cc:2369] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:37:23.504782 16113 raft_consensus.cc:482] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:23.505071 16113 raft_consensus.cc:504] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.506503 16113 leader_election.cc:288] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76df57f032824042879d9465e1c8f284 (127.2.193.130:34047), e6988e4e46df4bc4978729e460d86cc3 (127.2.193.131:33537)
I20260924 10:37:23.518373 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.507097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:51153 (local address 127.2.193.130:34047)
0924 10:37:23.507234 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:23.507239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:23.509501 (+ 2262us) server_negotiation.cc:407] Connection header received
0924 10:37:23.509707 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:23.509711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:23.509759 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:23.509807 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:23.510263 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.511071 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.512798 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.513445 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.513540 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.513592 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:23.518126 (+ 4534us) server_negotiation.cc:299] Negotiation successful
0924 10:37:23.518228 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:37:23.519076 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.506650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34047 (local address 127.2.193.129:51153)
0924 10:37:23.509410 (+ 2760us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:23.509426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:23.509585 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:23.509967 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:23.509970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:23.509988 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:23.510133 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.510138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.511218 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.511221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.512654 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.512666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.517844 (+ 5178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.517848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.517970 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.517998 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:37:23.518808 (+ 810us) client_negotiation.cc:240] Negotiation successful
0924 10:37:23.518895 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2678,"thread_start_us":141,"threads_started":1}
I20260924 10:37:23.519999 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33080143eaa7468cb9f3cca32551caa8" candidate_uuid: "f42c3cb88c0a44168ff8093d346ad162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76df57f032824042879d9465e1c8f284" is_pre_election: true
I20260924 10:37:23.520358 15896 raft_consensus.cc:2454] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f42c3cb88c0a44168ff8093d346ad162 in term 0.
I20260924 10:37:23.520955 15695 leader_election.cc:302] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76df57f032824042879d9465e1c8f284, f42c3cb88c0a44168ff8093d346ad162; no voters:
I20260924 10:37:23.521333 16113 raft_consensus.cc:2781] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:37:23.521414 16113 raft_consensus.cc:482] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:37:23.521458 16113 raft_consensus.cc:3037] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:23.522828 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.507127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:40645 (local address 127.2.193.131:33537)
0924 10:37:23.507358 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:23.507363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:23.513475 (+ 6112us) server_negotiation.cc:407] Connection header received
0924 10:37:23.513668 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:23.513671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:23.513723 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:23.513769 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:23.514159 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.514878 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.517715 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.518315 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.518414 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.518476 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:23.522595 (+ 4119us) server_negotiation.cc:299] Negotiation successful
0924 10:37:23.522699 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260924 10:37:23.522861 16113 raft_consensus.cc:504] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.523239 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.506743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:33537 (local address 127.2.193.129:40645)
0924 10:37:23.513378 (+ 6635us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:23.513392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:23.513549 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:23.513895 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:23.513898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:23.513913 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:23.514032 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.514037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.516316 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.516319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.517561 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:23.517573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.522305 (+ 4732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.522309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:23.522430 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.522466 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:37:23.523016 (+ 550us) client_negotiation.cc:240] Negotiation successful
0924 10:37:23.523100 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6562,"thread_start_us":101,"threads_started":1}
I20260924 10:37:23.523291 16113 leader_election.cc:288] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [CANDIDATE]: Term 1 election: Requested vote from peers 76df57f032824042879d9465e1c8f284 (127.2.193.130:34047), e6988e4e46df4bc4978729e460d86cc3 (127.2.193.131:33537)
I20260924 10:37:23.524376 16036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33080143eaa7468cb9f3cca32551caa8" candidate_uuid: "f42c3cb88c0a44168ff8093d346ad162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6988e4e46df4bc4978729e460d86cc3" is_pre_election: true
I20260924 10:37:23.524712 16036 raft_consensus.cc:2454] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f42c3cb88c0a44168ff8093d346ad162 in term 0.
I20260924 10:37:23.525043 16035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33080143eaa7468cb9f3cca32551caa8" candidate_uuid: "f42c3cb88c0a44168ff8093d346ad162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6988e4e46df4bc4978729e460d86cc3"
I20260924 10:37:23.525143 16035 raft_consensus.cc:3037] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:23.526108 15896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33080143eaa7468cb9f3cca32551caa8" candidate_uuid: "f42c3cb88c0a44168ff8093d346ad162" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76df57f032824042879d9465e1c8f284"
I20260924 10:37:23.526252 15896 raft_consensus.cc:3037] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:23.526521 16035 raft_consensus.cc:2454] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42c3cb88c0a44168ff8093d346ad162 in term 1.
I20260924 10:37:23.527146 15696 leader_election.cc:302] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6988e4e46df4bc4978729e460d86cc3, f42c3cb88c0a44168ff8093d346ad162; no voters:
I20260924 10:37:23.527355 16113 raft_consensus.cc:2781] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:23.527581 16113 raft_consensus.cc:686] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [term 1 LEADER]: Becoming Leader. State: Replica: f42c3cb88c0a44168ff8093d346ad162, State: Running, Role: LEADER
I20260924 10:37:23.527613 15896 raft_consensus.cc:2454] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42c3cb88c0a44168ff8093d346ad162 in term 1.
I20260924 10:37:23.527945 16113 consensus_queue.cc:237] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } }
I20260924 10:37:23.532193 15609 catalog_manager.cc:5495] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 reported cstate change: term changed from 0 to 1, leader changed from <none> to f42c3cb88c0a44168ff8093d346ad162 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "f42c3cb88c0a44168ff8093d346ad162" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42c3cb88c0a44168ff8093d346ad162" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 } health_report { overall_health: UNKNOWN } } }
I20260924 10:37:23.575593 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.570510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.2.193.129:45359)
0924 10:37:23.570632 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:23.570636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:23.570654 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:37:23.570742 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:23.570744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:23.570798 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:23.570956 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:37:23.571225 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.571984 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.573245 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:23.573603 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:23.573701 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:23.574181 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:37:23.574758 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:37:23.574872 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:23.575348 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 10:37:23.575416 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:37:23.593792 16035 raft_consensus.cc:1261] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 1 FOLLOWER]: Refusing update from remote peer f42c3cb88c0a44168ff8093d346ad162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:23.594753 15896 raft_consensus.cc:1261] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 1 FOLLOWER]: Refusing update from remote peer f42c3cb88c0a44168ff8093d346ad162: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:23.595815 16113 consensus_queue.cc:1035] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6988e4e46df4bc4978729e460d86cc3" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:23.596272 16113 consensus_queue.cc:1035] T 33080143eaa7468cb9f3cca32551caa8 P f42c3cb88c0a44168ff8093d346ad162 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76df57f032824042879d9465e1c8f284" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:23.617095 16124 mvcc.cc:205] Tried to move back new op lower bound from 7332848613750710272 to 7332848613493657600. Current Snapshot: MvccSnapshot[applied={T|T < 7332848613750710272}]
I20260924 10:37:23.622367 16125 mvcc.cc:205] Tried to move back new op lower bound from 7332848613750710272 to 7332848613493657600. Current Snapshot: MvccSnapshot[applied={T|T < 7332848613750710272}]
I20260924 10:37:23.635699 15606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:23.635915 15606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:23.638955 15606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c59fbb1134524e139b41eab5fe297f14: Sending DeleteTablet for 3 replicas of tablet 33080143eaa7468cb9f3cca32551caa8
I20260924 10:37:23.640251 15857 tablet_service.cc:1514] Processing DeleteTablet for tablet 33080143eaa7468cb9f3cca32551caa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:23 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:43958
I20260924 10:37:23.640758 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15647
I20260924 10:37:23.640969 16016 tablet_service.cc:1514] Processing DeleteTablet for tablet 33080143eaa7468cb9f3cca32551caa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:23 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:43338
I20260924 10:37:23.643684 16133 tablet_replica.cc:330] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: stopping tablet replica
I20260924 10:37:23.644058 16133 raft_consensus.cc:2229] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:23.644414 16133 raft_consensus.cc:2258] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:23.645979 16133 ts_tablet_manager.cc:1905] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:23.649140 16133 ts_tablet_manager.cc:1918] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:23.649771 16134 tablet_replica.cc:330] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284: stopping tablet replica
W20260924 10:37:23.649819 15594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:45359 (error 108)
I20260924 10:37:23.650240 16134 raft_consensus.cc:2229] T 33080143eaa7468cb9f3cca32551caa8 P 76df57f032824042879d9465e1c8f284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:23.650579 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15811
I20260924 10:37:23.657335 16133 log.cc:1196] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/33080143eaa7468cb9f3cca32551caa8
I20260924 10:37:23.657842 16133 ts_tablet_manager.cc:1939] T 33080143eaa7468cb9f3cca32551caa8 P e6988e4e46df4bc4978729e460d86cc3: Deleting consensus metadata
I20260924 10:37:23.658427 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15945
I20260924 10:37:23.660750 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.660408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:45359 (local address 127.0.0.1:51112)
0924 10:37:23.660571 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:23.660646 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:45359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:37:23.661056 16105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:23.660633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34047 (local address 127.0.0.1:43972)
0924 10:37:23.660920 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:23.660987 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:34047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
W20260924 10:37:23.662000 15594 catalog_manager.cc:4584] TS f42c3cb88c0a44168ff8093d346ad162 (127.2.193.129:45359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33080143eaa7468cb9f3cca32551caa8: Network error: Client connection negotiation failed: client connection to 127.2.193.129:45359: connect: Connection refused (error 111)
I20260924 10:37:23.666246 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 15576
I20260924 10:37:23.675060 2822 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:37:24.020826 2822 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:37:24.020999 2822 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:37:24.366689 2822 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:37:24.366 UTC [14643] LOG: received smart shutdown request
2026-09-24 10:37:24.371 UTC [14643] LOG: background worker "logical replication launcher" (PID 14652) exited with exit code 1
2026-09-24 10:37:24.371 UTC [14647] LOG: shutting down
2026-09-24 10:37:24.542 UTC [14643] LOG: database system is shut down
2026-09-24T10:37:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49492 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:37:24 dist-test-slave-8wc7 krb5kdc[16144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:37:24 dist-test-slave-8wc7 krb5kdc[16144](info): set up 2 sockets
Sep 24 10:37:24 dist-test-slave-8wc7 krb5kdc[16144](info): commencing operation
krb5kdc: starting...
W20260924 10:37:25.701213 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:37:25 dist-test-slave-8wc7 krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:37:25.827725 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:39969
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:39969,127.2.193.189:39935,127.2.193.188:41863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:25.939213 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:25.939519 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:25.939564 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:25.943568 16157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:25.943632 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:25.943651 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:25.943668 16157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:25.943683 16157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:25.949224 16157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:39969,127.2.193.189:39935,127.2.193.188:41863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:39969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:25.950314 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:25.951356 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:25.960294 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:25.960409 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:25.960409 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:25.960845 16157 server_base.cc:1034] running on GCE node
I20260924 10:37:25.961493 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:25.962636 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:25.963796 16157 hybrid_clock.cc:648] HybridClock initialized: now 1790246245963766 us; error 52 us; skew 500 ppm
Sep 24 10:37:25 dist-test-slave-8wc7 krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:25.966794 16157 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:25.967741 16157 webserver.cc:460] Webserver started at http://127.2.193.190:32943/ using document root <none> and password file <none>
I20260924 10:37:25.968302 16157 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:25.968396 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:25.968657 16157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:25.970799 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "77bc3e81d6524aa1909620d11a40e450"
format_stamp: "Formatted at 2026-09-24 10:37:25 on dist-test-slave-8wc7"
server_key: "2fabfc9bfca2a7b56e94724840af3372"
server_key_iv: "49006de24c3ca0756239998ff431e6c3"
server_key_version: "encryptionkey@0"
I20260924 10:37:25.971361 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "77bc3e81d6524aa1909620d11a40e450"
format_stamp: "Formatted at 2026-09-24 10:37:25 on dist-test-slave-8wc7"
server_key: "2fabfc9bfca2a7b56e94724840af3372"
server_key_iv: "49006de24c3ca0756239998ff431e6c3"
server_key_version: "encryptionkey@0"
I20260924 10:37:25.975167 16157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:37:25.977687 16172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:25.978927 16157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:37:25.979075 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "77bc3e81d6524aa1909620d11a40e450"
format_stamp: "Formatted at 2026-09-24 10:37:25 on dist-test-slave-8wc7"
server_key: "2fabfc9bfca2a7b56e94724840af3372"
server_key_iv: "49006de24c3ca0756239998ff431e6c3"
server_key_version: "encryptionkey@0"
I20260924 10:37:25.979189 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:25.992833 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:25.996265 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:25.996507 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:26.003818 16157 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:39969
I20260924 10:37:26.003834 16224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:39969 every 8 connection(s)
I20260924 10:37:26.004807 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:26.007663 16225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:26.011565 16225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.014420 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16157
I20260924 10:37:26.014547 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:37:26.014825 2822 external_mini_cluster.cc:1428] Setting key 0581d6b1d6888d9f44be58626a851958
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.189@KRBTEST.COM" created.
I20260924 10:37:26.028578 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.013935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39969 (local address 127.0.0.1:57586)
0924 10:37:26.014448 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.014502 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.015417 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.016267 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.016276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.016797 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.017232 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.017254 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.018405 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.018412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.019738 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.019751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.020424 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.020431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.020595 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.021257 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.021282 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.023005 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.025157 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.025166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.025293 (+ 127us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.025633 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.025995 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.025998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.026000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.026131 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.026868 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.026879 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.027069 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.027266 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.027729 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":70,"thread_start_us":158,"threads_started":1}
I20260924 10:37:26.028584 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.013935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.2.193.190:39969)
0924 10:37:26.014549 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.014561 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.014608 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:37:26.015749 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.015759 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.016019 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.016232 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.017395 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.018259 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.019867 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.020244 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.020368 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.023165 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.023171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.023182 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.023203 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.025022 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.025757 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.025764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.025766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.025895 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.026267 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.026273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.026278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.026731 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.026844 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.027376 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.027623 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":486,"spinlock_wait_cycles":40320,"thread_start_us":179,"threads_started":1}
I20260924 10:37:26.031162 16225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.031936 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.031669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35216)
0924 10:37:26.031795 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.031875 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:37:26.032138 16173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
I20260924 10:37:26.032967 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.032763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35232)
0924 10:37:26.032868 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.032918 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:37:26.033123 16225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:39935: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab.
Entry for principal kudu/127.2.193.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab.
I20260924 10:37:26.044271 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:39935
--webserver_interface=127.2.193.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab
--principal=kudu/127.2.193.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:39969,127.2.193.189:39935,127.2.193.188:41863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:37:26.082419 16225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } attempt: 1
I20260924 10:37:26.083370 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.083027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35240)
0924 10:37:26.083198 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.083275 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:37:26.084169 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.083905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35242)
0924 10:37:26.084029 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.084105 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:37:26.084358 16225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:39935: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
I20260924 10:37:26.152583 16225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } attempt: 2
I20260924 10:37:26.153625 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.153225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35246)
0924 10:37:26.153458 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.153539 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 10:37:26.154457 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.154204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35256)
0924 10:37:26.154337 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.154391 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 10:37:26.154603 16225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:39935: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
W20260924 10:37:26.159047 16232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:26.159358 16232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:26.159458 16232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:26.163695 16232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:26.163785 16232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:26.163821 16232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:26.163878 16232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:26.163916 16232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:26.169584 16232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:39969,127.2.193.189:39935,127.2.193.188:41863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab
--principal=kudu/127.2.193.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:39935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.193.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:26.170778 16232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:26.171871 16232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:26.181128 16237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:26.181142 16240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:26.181146 16238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:26.181363 16232 server_base.cc:1034] running on GCE node
I20260924 10:37:26.182107 16232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:26.183429 16232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:26.184602 16232 hybrid_clock.cc:648] HybridClock initialized: now 1790246246184558 us; error 84 us; skew 500 ppm
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:26.187489 16232 init.cc:377] Logged in from keytab as kudu/127.2.193.189@KRBTEST.COM (short username kudu)
I20260924 10:37:26.188468 16232 webserver.cc:460] Webserver started at http://127.2.193.189:36145/ using document root <none> and password file <none>
I20260924 10:37:26.189055 16232 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:26.189146 16232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:26.189424 16232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:26.191579 16232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data/instance:
uuid: "a5e81db00da743a5b61b08422b6c08c7"
format_stamp: "Formatted at 2026-09-24 10:37:26 on dist-test-slave-8wc7"
server_key: "49f4a4dd26aaccd260f3111c224dac1c"
server_key_iv: "e9cb551081bc2cc8cc0334b6e19a25cc"
server_key_version: "encryptionkey@0"
I20260924 10:37:26.192188 16232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/wal/instance:
uuid: "a5e81db00da743a5b61b08422b6c08c7"
format_stamp: "Formatted at 2026-09-24 10:37:26 on dist-test-slave-8wc7"
server_key: "49f4a4dd26aaccd260f3111c224dac1c"
server_key_iv: "e9cb551081bc2cc8cc0334b6e19a25cc"
server_key_version: "encryptionkey@0"
I20260924 10:37:26.196200 16232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 10:37:26.198837 16247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:26.200083 16232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:37:26.200232 16232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/wal
uuid: "a5e81db00da743a5b61b08422b6c08c7"
format_stamp: "Formatted at 2026-09-24 10:37:26 on dist-test-slave-8wc7"
server_key: "49f4a4dd26aaccd260f3111c224dac1c"
server_key_iv: "e9cb551081bc2cc8cc0334b6e19a25cc"
server_key_version: "encryptionkey@0"
I20260924 10:37:26.200352 16232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:26.240664 16232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:26.245287 16232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:26.245515 16232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:26.245774 16225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } attempt: 3
I20260924 10:37:26.246886 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.246444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35264)
0924 10:37:26.246621 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.246691 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 10:37:26.248968 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.247839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35280)
0924 10:37:26.248834 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.248901 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912}
W20260924 10:37:26.249135 16225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:39935: Network error: Client connection negotiation failed: client connection to 127.2.193.189:39935: connect: Connection refused (error 111)
I20260924 10:37:26.253975 16232 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.189:39935
I20260924 10:37:26.253983 16299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.189:39935 every 8 connection(s)
I20260924 10:37:26.254974 16232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
I20260924 10:37:26.257954 16300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:26.261770 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16232
I20260924 10:37:26.261893 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-1/wal/instance
I20260924 10:37:26.261752 16300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.262161 2822 external_mini_cluster.cc:1428] Setting key 63de8ef70c80e6f84ad93b3608678636
WARNING: no policy specified for kudu/127.2.193.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.188@KRBTEST.COM" created.
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:26.277024 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.263929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.2.193.190:39969)
0924 10:37:26.264109 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.264115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.264937 (+ 822us) server_negotiation.cc:407] Connection header received
0924 10:37:26.265940 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.265944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.266016 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.266109 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.267234 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.267769 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.268883 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.269134 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.269293 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.273148 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.273151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.273153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.273217 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.274701 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.275230 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.275237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.275241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.275337 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.275658 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.275665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.275669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.275952 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.276072 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.276585 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.276740 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:37:26.277631 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.264332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39969 (local address 127.0.0.1:57596)
0924 10:37:26.264790 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.264842 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.265725 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.266278 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.266285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.266622 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.267092 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.267105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.267899 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.267904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.268760 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.268770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.269337 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.269341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.269517 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.270147 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.270163 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.272966 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.274812 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.274817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.274828 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.275116 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.275434 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.275436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.275437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.275563 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.276083 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.276087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.276322 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.276488 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.276737 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":133,"threads_started":1}
I20260924 10:37:26.279120 16300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab.
Entry for principal kudu/127.2.193.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab.
I20260924 10:37:26.287951 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.279625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35286)
0924 10:37:26.279740 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.279756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.279824 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.280369 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.280372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.280531 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.280609 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.280614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.281609 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.281612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.282145 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.282153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.282724 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.282727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.282836 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.283300 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.283315 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.284579 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.286431 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.286436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.286438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.286628 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.286908 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.286911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.286913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.286991 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.287441 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.287444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.287533 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.287640 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.287762 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260924 10:37:26.288017 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.279768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.2.193.189:39935)
0924 10:37:26.280048 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.280056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.280073 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:37:26.280104 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.280108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.280255 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.280394 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.280713 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.281494 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.282257 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.282616 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.282726 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.284702 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.284711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.284725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.284753 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.286308 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.286731 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.286736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.286738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.286814 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.287079 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.287082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.287084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.287318 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.287465 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.287728 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.287848 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260924 10:37:26.288731 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:41863
--webserver_interface=127.2.193.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab
--principal=kudu/127.2.193.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:39969,127.2.193.189:39935,127.2.193.188:41863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:37:26.290215 16300 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.290931 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.290674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40480)
0924 10:37:26.290791 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.290871 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 10:37:26.291162 16249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
I20260924 10:37:26.292062 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.291773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40490)
0924 10:37:26.291908 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.291979 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:37:26.292210 16300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:41863: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
I20260924 10:37:26.341512 16300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } attempt: 1
I20260924 10:37:26.342470 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.342109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40500)
0924 10:37:26.342298 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.342373 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 10:37:26.343222 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.343008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40504)
0924 10:37:26.343115 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.343167 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:37:26.343351 16300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:41863: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
I20260924 10:37:26.392374 16225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } attempt: 4
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
I20260924 10:37:26.402854 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.393153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.2.193.189:39935)
0924 10:37:26.393316 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.393319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.393381 (+ 62us) server_negotiation.cc:407] Connection header received
0924 10:37:26.393522 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.393525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.393569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.393645 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.394683 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.395572 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.396710 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.396950 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.397036 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.399457 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.399460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.399462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.399490 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.400950 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.401462 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.401465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.401466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.401542 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.401894 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.401898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.401900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.402197 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.402283 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.402622 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.402731 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 10:37:26.402863 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.392990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35292)
0924 10:37:26.393276 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.393295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.393421 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.393701 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.393704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.393921 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.394523 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.394535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.395707 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.395711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.396581 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.396593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.397080 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.397084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.397234 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.397749 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.397768 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.399294 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.401084 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.401088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.401090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.401366 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.401676 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.401679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.401681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.401773 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.402342 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.402345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.402441 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.402537 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.402686 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 10:37:26.403798 16225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.404651 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.404321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40508)
0924 10:37:26.404481 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.404564 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:37:26.405421 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.405219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40518)
0924 10:37:26.405310 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.405358 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 10:37:26.405583 16225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:41863: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
W20260924 10:37:26.408066 16306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:26.408418 16306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:26.408502 16306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:37:26.411520 16300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } attempt: 2
I20260924 10:37:26.412304 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.411993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40520)
0924 10:37:26.412151 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.412216 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:37:26.413021 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.412803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40534)
0924 10:37:26.412913 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.412961 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:37:26.413194 16300 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:41863: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
W20260924 10:37:26.413455 16306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:26.413506 16306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:26.413547 16306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:26.413579 16306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:26.413611 16306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:26.419418 16306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:39969,127.2.193.189:39935,127.2.193.188:41863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab
--principal=kudu/127.2.193.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:41863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.193.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:26.420596 16306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:26.421800 16306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:26.430910 16313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:26.430961 16312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:26.431159 16306 server_base.cc:1034] running on GCE node
W20260924 10:37:26.430910 16315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:26.431938 16306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:26.433156 16306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:26.434409 16306 hybrid_clock.cc:648] HybridClock initialized: now 1790246246434393 us; error 33 us; skew 500 ppm
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:26.437525 16306 init.cc:377] Logged in from keytab as kudu/127.2.193.188@KRBTEST.COM (short username kudu)
I20260924 10:37:26.438505 16306 webserver.cc:460] Webserver started at http://127.2.193.188:34455/ using document root <none> and password file <none>
I20260924 10:37:26.439091 16306 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:26.439158 16306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:26.439337 16306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:26.441569 16306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data/instance:
uuid: "98df876ff9004a82a421c0bcd1d4b50e"
format_stamp: "Formatted at 2026-09-24 10:37:26 on dist-test-slave-8wc7"
server_key: "94d2360fcf86a17d944ab4d01c06b7d8"
server_key_iv: "ad3458ce24b100b1c064b4fabebdb4a6"
server_key_version: "encryptionkey@0"
I20260924 10:37:26.442134 16306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/wal/instance:
uuid: "98df876ff9004a82a421c0bcd1d4b50e"
format_stamp: "Formatted at 2026-09-24 10:37:26 on dist-test-slave-8wc7"
server_key: "94d2360fcf86a17d944ab4d01c06b7d8"
server_key_iv: "ad3458ce24b100b1c064b4fabebdb4a6"
server_key_version: "encryptionkey@0"
I20260924 10:37:26.446158 16306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:37:26.448853 16322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:26.450246 16306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:37:26.450400 16306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/wal
uuid: "98df876ff9004a82a421c0bcd1d4b50e"
format_stamp: "Formatted at 2026-09-24 10:37:26 on dist-test-slave-8wc7"
server_key: "94d2360fcf86a17d944ab4d01c06b7d8"
server_key_iv: "ad3458ce24b100b1c064b4fabebdb4a6"
server_key_version: "encryptionkey@0"
I20260924 10:37:26.450546 16306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:26.464761 16225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } attempt: 1
I20260924 10:37:26.465720 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.465394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40538)
0924 10:37:26.465560 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.465629 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:37:26.466467 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.466255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40548)
0924 10:37:26.466363 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.466413 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:37:26.466622 16225 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:41863: Network error: Client connection negotiation failed: client connection to 127.2.193.188:41863: connect: Connection refused (error 111)
I20260924 10:37:26.473352 16306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:26.476999 16306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:26.477284 16306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:26.484994 16306 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.188:41863
I20260924 10:37:26.485015 16374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.188:41863 every 8 connection(s)
I20260924 10:37:26.486090 16306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
I20260924 10:37:26.489110 16375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:26.493312 16375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.495846 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16306
I20260924 10:37:26.496001 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246085895225-2822-0/minicluster-data/master-2/wal/instance
I20260924 10:37:26.496284 2822 external_mini_cluster.cc:1428] Setting key bef81c25e5ac8b57be609efa362c9df2
I20260924 10:37:26.504390 16300 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } attempt: 3
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
I20260924 10:37:26.510609 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.495746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.2.193.190:39969)
0924 10:37:26.495885 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.495888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.497210 (+ 1322us) server_negotiation.cc:407] Connection header received
0924 10:37:26.498303 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.498307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.498361 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.498485 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.499617 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.500509 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.501978 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.502382 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.502547 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.506623 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.506629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.506635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.506671 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.508289 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.508933 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.508937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.508939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.509030 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.509425 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.509429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.509431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.509638 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.509748 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.510297 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.510422 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
I20260924 10:37:26.511660 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.496337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39969 (local address 127.0.0.1:57612)
0924 10:37:26.496982 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.497053 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.498084 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.498583 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.498588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.499073 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.499450 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.499480 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.500632 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.500634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.501817 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.501828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.502590 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.502593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.502797 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.503580 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.503602 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.506448 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.508408 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.508420 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.508432 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.508819 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.509123 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.509128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.509130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.509322 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.509762 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.509771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.510048 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.510240 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.510580 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":46,"spinlock_wait_cycles":768,"thread_start_us":171,"threads_started":1}
I20260924 10:37:26.512622 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.497470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.2.193.188:41863)
0924 10:37:26.497753 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.497765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.497809 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:37:26.498097 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.498119 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.498380 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.498587 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.499462 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.500838 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.501978 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.502647 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.502829 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.506881 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.506889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.506906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.506932 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.509792 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.510392 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.510399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.510401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.510500 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.510880 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.510884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.510886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.511214 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.511438 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.511892 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.512061 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
I20260924 10:37:26.513509 16375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.516010 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.504964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40570)
0924 10:37:26.505104 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.505116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.505242 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.506018 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.506021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.506183 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.506271 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.506276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.507361 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.507365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.507959 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.507969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.508681 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.508685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.508787 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.509994 (+ 1207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.510013 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.511632 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.514108 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.514111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.514114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.514374 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.514990 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.514996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.515002 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.515098 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.515543 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.515546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.515638 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.515718 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.515845 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:37:26.516759 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.505309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.2.193.188:41863)
0924 10:37:26.505731 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.505740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.505756 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:37:26.505795 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.505798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.505843 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.505966 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.506394 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.507211 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.508090 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.508540 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.508663 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.511789 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.511795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.511799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.511828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.513985 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.514801 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.514804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.514806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.514884 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.515210 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.515214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.515217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.515415 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.516419 (+ 1004us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.516497 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.516600 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":167,"threads_started":1}
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
I20260924 10:37:26.520371 16300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: Bootstrap starting.
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:26.523319 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.514139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.2.193.189:39935)
0924 10:37:26.514509 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.514512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.514520 (+ 8us) server_negotiation.cc:407] Connection header received
0924 10:37:26.514543 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.514545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.514576 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.514663 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.515079 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.516176 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.517267 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.517530 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.517627 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.519724 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.519728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.519730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.519753 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.521498 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.522023 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.522029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.522034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.522116 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.522387 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.522392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.522393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.522602 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.522715 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.523015 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.523138 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260924 10:37:26.523386 16300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:26.523761 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.514044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:39935 (local address 127.0.0.1:35294)
0924 10:37:26.514213 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.514227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.514297 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.514678 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.514681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.514848 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.514951 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.514959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.516305 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.516308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.517095 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.517104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.517723 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.517726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.517790 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.518313 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.518327 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.519590 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.521641 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.521646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.521651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.521896 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.522222 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.522224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.522225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.522290 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.522705 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.522708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.522815 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.523495 (+ 680us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.523635 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:37:26.524501 16300 log.cc:824] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:26.524559 16375 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:37:26.527305 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.515745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.2.193.190:39969)
0924 10:37:26.516286 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.516291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.516304 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:37:26.516479 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.516483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.516534 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.516632 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.517893 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.518622 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.519817 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.520139 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.520260 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.523350 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.523354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.523357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.523526 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.525527 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.526040 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.526043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.526045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.526138 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.526428 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.526432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.526433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.526629 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.526863 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.527036 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.527152 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446}
I20260924 10:37:26.527673 16300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: No bootstrap required, opened a new log
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246246, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
I20260924 10:37:26.532090 16300 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.532346 16300 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:26.532429 16300 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e81db00da743a5b61b08422b6c08c7, State: Initialized, Role: FOLLOWER
I20260924 10:37:26.533051 16300 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.533812 16225 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } attempt: 2
I20260924 10:37:26.534539 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.525101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40574)
0924 10:37:26.525210 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.525225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.525299 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.525559 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.525560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.525754 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.525861 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.525868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.526949 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.526955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.527748 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.527758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.528336 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.528340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.528453 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.528872 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.528888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.530507 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.532449 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.532452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.532455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.532683 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.532955 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.532959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.532961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.533065 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.533632 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.533634 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.533722 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.534170 (+ 448us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.534327 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 10:37:26.534752 16300 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:26.535113 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.525164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.2.193.188:41863)
0924 10:37:26.525325 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.525331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.525346 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:37:26.525387 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.525391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.525454 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.525555 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.526002 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.526840 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.527874 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.528211 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.528318 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.530653 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.530657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.530659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.530683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.532340 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.532780 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.532783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.532785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.532855 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.533205 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.533208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.533210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.533507 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.533600 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.534815 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.534953 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
I20260924 10:37:26.537406 16382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.537760 16382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:37:26.539465 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e: Bootstrap starting.
Sep 24 10:37:26 dist-test-slave-8wc7 krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246245, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
W20260924 10:37:26.542045 16393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:37:26.542119 16393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:37:26.539551 16232 master_runner.cc:186] Error getting master registration for 127.2.193.190:39969: OK, instance_id { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" instance_seqno: 1790246246001737 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:37:26.543489 16232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:37:26.544571 16375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:26.545688 16375 log.cc:824] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:26.546679 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.534225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:41863 (local address 127.0.0.1:40586)
0924 10:37:26.534512 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:26.534529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:26.534611 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:26.535483 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:26.535486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:26.535716 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:26.535862 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.535875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.537565 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.537571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.538295 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:26.538307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.538884 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.538888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:26.538986 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.539483 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:26.539503 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:26.542326 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:26.544719 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.544723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.544726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.544969 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.545374 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:26.545378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:26.545380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:26.545492 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:26.546053 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:26.546057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:26.546174 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 10:37:26.546290 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:37:26.546476 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260924 10:37:26.547500 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.529522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.2.193.189:39935)
0924 10:37:26.529637 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.529641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.529653 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:37:26.529694 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.529695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.529727 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.529823 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.530496 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.531796 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.532978 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.533320 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.533417 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.536564 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.536568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.536571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.536600 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.543968 (+ 7368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.545810 (+ 1842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.545814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.545816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.545921 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.546296 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.546300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.546302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.546527 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.546709 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.547168 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.547308 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:37:26.547601 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.534758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.2.193.188:41863)
0924 10:37:26.535162 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.535167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.535184 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:37:26.535221 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.535224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.535267 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.535346 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.536253 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.537146 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.538424 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.538743 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.540049 (+ 1306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.542510 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.542514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.542517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.542547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.544572 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.545088 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.545095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.545097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.545248 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.545617 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.545624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.545628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.545891 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.545997 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.547291 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.547445 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260924 10:37:26.548148 16375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e: No bootstrap required, opened a new log
I20260924 10:37:26.551005 16225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: Bootstrap starting.
I20260924 10:37:26.552416 16375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.553186 16375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:26.553301 16375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98df876ff9004a82a421c0bcd1d4b50e, State: Initialized, Role: FOLLOWER
I20260924 10:37:26.553913 16375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.554064 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:37:26.554320 16225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:26.555413 16225 log.cc:824] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:26.555518 16375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:26.557860 16225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: No bootstrap required, opened a new log
W20260924 10:37:26.560986 16412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:37:26.561120 16412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:37:26.558892 16306 master_runner.cc:186] Error getting master registration for 127.2.193.190:39969: OK, instance_id { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" instance_seqno: 1790246246001737 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:37:26.563112 16306 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:37:26.562963 16225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.563356 16225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:26.563483 16225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bc3e81d6524aa1909620d11a40e450, State: Initialized, Role: FOLLOWER
I20260924 10:37:26.564153 16225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.565816 16225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:26.569000 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.569152 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:37:26.569203 16416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.569284 16396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:26.569393 16416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:37:26.569424 16396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.570909 16199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98df876ff9004a82a421c0bcd1d4b50e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bc3e81d6524aa1909620d11a40e450" is_pre_election: true
I20260924 10:37:26.571187 16199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98df876ff9004a82a421c0bcd1d4b50e in term 0.
I20260924 10:37:26.572008 16326 leader_election.cc:302] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bc3e81d6524aa1909620d11a40e450, 98df876ff9004a82a421c0bcd1d4b50e; no voters:
I20260924 10:37:26.572803 16396 leader_election.cc:288] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bc3e81d6524aa1909620d11a40e450 (127.2.193.190:39969), a5e81db00da743a5b61b08422b6c08c7 (127.2.193.189:39935)
I20260924 10:37:26.573617 16274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98df876ff9004a82a421c0bcd1d4b50e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e81db00da743a5b61b08422b6c08c7" is_pre_election: true
I20260924 10:37:26.573927 16274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98df876ff9004a82a421c0bcd1d4b50e in term 0.
I20260924 10:37:26.573757 16418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:37:26.575443 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.562312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.2.193.188:41863)
0924 10:37:26.562431 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.562435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.563735 (+ 1300us) server_negotiation.cc:407] Connection header received
0924 10:37:26.564509 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.564512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.564562 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.564611 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.566048 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.566806 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.567964 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.568307 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.568376 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.571436 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.571440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.571443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.571469 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.573600 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.574080 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.574083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.574086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.574171 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.574457 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.574460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.574461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.574672 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.574794 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.575148 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.575281 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:37:26.576262 16418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:37:26.576354 16418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:26.577687 16418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.578112 16418 leader_election.cc:288] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [CANDIDATE]: Term 1 election: Requested vote from peers 77bc3e81d6524aa1909620d11a40e450 (127.2.193.190:39969), a5e81db00da743a5b61b08422b6c08c7 (127.2.193.189:39935)
I20260924 10:37:26.578409 16199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98df876ff9004a82a421c0bcd1d4b50e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bc3e81d6524aa1909620d11a40e450"
I20260924 10:37:26.578550 16199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:26.579859 16199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98df876ff9004a82a421c0bcd1d4b50e in term 1.
I20260924 10:37:26.580245 16326 leader_election.cc:302] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bc3e81d6524aa1909620d11a40e450, 98df876ff9004a82a421c0bcd1d4b50e; no voters:
I20260924 10:37:26.580730 16418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:26.580860 16418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [term 1 LEADER]: Becoming Leader. State: Replica: 98df876ff9004a82a421c0bcd1d4b50e, State: Running, Role: LEADER
I20260924 10:37:26.581386 16418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } }
I20260924 10:37:26.581720 16274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98df876ff9004a82a421c0bcd1d4b50e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e81db00da743a5b61b08422b6c08c7"
I20260924 10:37:26.581871 16274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:26.583455 16274 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98df876ff9004a82a421c0bcd1d4b50e in term 1.
I20260924 10:37:26.585655 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98df876ff9004a82a421c0bcd1d4b50e. Latest consensus state: current_term: 1 leader_uuid: "98df876ff9004a82a421c0bcd1d4b50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.585789 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:26.586114 16157 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260924 10:37:26.586278 16157 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:37:26.588546 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.561701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.2.193.190:39969)
0924 10:37:26.561851 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.561856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.562764 (+ 908us) server_negotiation.cc:407] Connection header received
0924 10:37:26.562995 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.562999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.563057 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.563115 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.564281 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.565111 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.566421 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.566829 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.566950 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.568920 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.568926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.568929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.568962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.578309 (+ 9347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.586425 (+ 8116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.586430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.586432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.586559 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.586979 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.586983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.586985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.587244 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.587411 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.588208 (+ 797us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.588350 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:37:26.590363 16431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:26.591416 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:26.564038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.2.193.189:39935)
0924 10:37:26.564152 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:26.564156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:26.572199 (+ 8043us) server_negotiation.cc:407] Connection header received
0924 10:37:26.572405 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:26.572409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:26.572471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:26.572528 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:26.573908 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.574690 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.585329 (+ 10639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:26.585713 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:26.585796 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:26.587762 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:26.587766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:26.587769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:26.587801 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:26.589566 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.590083 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.590087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.590089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.590172 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:26.590452 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:26.590455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:26.590457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:26.590684 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:26.590795 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:26.591139 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 10:37:26.591263 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:37:26.591504 16431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:26.597973 16199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [term 1 FOLLOWER]: Refusing update from remote peer 98df876ff9004a82a421c0bcd1d4b50e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:26.598026 16274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [term 1 FOLLOWER]: Refusing update from remote peer 98df876ff9004a82a421c0bcd1d4b50e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:26.599699 16418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:26.600198 16418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:26.605036 16382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98df876ff9004a82a421c0bcd1d4b50e. Latest consensus state: current_term: 1 leader_uuid: "98df876ff9004a82a421c0bcd1d4b50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.605252 16382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:37:26.605458 16416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98df876ff9004a82a421c0bcd1d4b50e. Latest consensus state: current_term: 1 leader_uuid: "98df876ff9004a82a421c0bcd1d4b50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.605602 16416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 10:37:26.606223 16429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:37:26.606325 16429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:37:26.612654 16432 mvcc.cc:205] Tried to move back new op lower bound from 7332848626055667712 to 7332848626001166336. Current Snapshot: MvccSnapshot[applied={T|T < 7332848626055667712}]
I20260924 10:37:26.613116 16396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98df876ff9004a82a421c0bcd1d4b50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.613260 16396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:26.614383 16418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98df876ff9004a82a421c0bcd1d4b50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.614503 16418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:26.614907 16382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98df876ff9004a82a421c0bcd1d4b50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.615038 16382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:37:26.615770 16435 mvcc.cc:205] Tried to move back new op lower bound from 7332848626055667712 to 7332848626001166336. Current Snapshot: MvccSnapshot[applied={T|T < 7332848626055667712}]
I20260924 10:37:26.616993 16416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98df876ff9004a82a421c0bcd1d4b50e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bc3e81d6524aa1909620d11a40e450" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39969 } } peers { permanent_uuid: "a5e81db00da743a5b61b08422b6c08c7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 39935 } } peers { permanent_uuid: "98df876ff9004a82a421c0bcd1d4b50e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 41863 } } }
I20260924 10:37:26.617102 16416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:37:26.618363 16436 mvcc.cc:205] Tried to move back new op lower bound from 7332848626055667712 to 7332848626001166336. Current Snapshot: MvccSnapshot[applied={T|T < 7332848626055667712}]
I20260924 10:37:26.623081 16431 catalog_manager.cc:1347] Generated new cluster ID: 88c599e87a73438b80f04ebff219c5c9
I20260924 10:37:26.623154 16431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:37:26.661324 16431 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:37:26.662832 16431 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:37:26.676882 16431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98df876ff9004a82a421c0bcd1d4b50e: Generated new TSK 0
I20260924 10:37:26.678049 16431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:37:27.543394 16393 catalog_manager.cc:1259] Loaded cluster ID: 88c599e87a73438b80f04ebff219c5c9
I20260924 10:37:27.543481 16393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: loading cluster ID for follower catalog manager: success
I20260924 10:37:27.545238 16393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: acquiring CA information for follower catalog manager: success
I20260924 10:37:27.546440 16393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5e81db00da743a5b61b08422b6c08c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:37:27.607745 16429 catalog_manager.cc:1259] Loaded cluster ID: 88c599e87a73438b80f04ebff219c5c9
I20260924 10:37:27.607848 16429 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: loading cluster ID for follower catalog manager: success
I20260924 10:37:27.609678 16429 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: acquiring CA information for follower catalog manager: success
I20260924 10:37:27.610926 16429 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77bc3e81d6524aa1909620d11a40e450: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:37:27.744372 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16157
I20260924 10:37:27.751190 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16232
I20260924 10:37:27.757359 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16306
2026-09-24T10:37:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3096 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:37:27 dist-test-slave-8wc7 krb5kdc[16449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:37:27 dist-test-slave-8wc7 krb5kdc[16449](info): set up 2 sockets
Sep 24 10:37:27 dist-test-slave-8wc7 krb5kdc[16449](info): commencing operation
krb5kdc: starting...
W20260924 10:37:28.805632 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:37:28 dist-test-slave-8wc7 krb5kdc[16449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:37:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:37:28.932945 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:43821
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:29.046608 16462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:29.046916 16462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:29.046958 16462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:29.051138 16462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:29.051210 16462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:29.051230 16462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:29.051249 16462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:29.051263 16462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:29.056864 16462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:29.058054 16462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:29.059072 16462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:29.067914 16468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:29.067929 16467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:29.067988 16462 server_base.cc:1034] running on GCE node
W20260924 10:37:29.067912 16470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:29.068871 16462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:29.070195 16462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:29.071386 16462 hybrid_clock.cc:648] HybridClock initialized: now 1790246249071374 us; error 29 us; skew 500 ppm
Sep 24 10:37:29 dist-test-slave-8wc7 krb5kdc[16449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:29.074342 16462 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:29.075289 16462 webserver.cc:460] Webserver started at http://127.2.193.190:40569/ using document root <none> and password file <none>
I20260924 10:37:29.075886 16462 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:29.075990 16462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:29.076229 16462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:29.078452 16462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "bb1d737edc304e358afb4f04aa620596"
format_stamp: "Formatted at 2026-09-24 10:37:29 on dist-test-slave-8wc7"
server_key: "e05e872bfd5b576ed117ab841a00786f"
server_key_iv: "f5c11af4271f77a2c1fdaf94cca26ef0"
server_key_version: "encryptionkey@0"
I20260924 10:37:29.079053 16462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "bb1d737edc304e358afb4f04aa620596"
format_stamp: "Formatted at 2026-09-24 10:37:29 on dist-test-slave-8wc7"
server_key: "e05e872bfd5b576ed117ab841a00786f"
server_key_iv: "f5c11af4271f77a2c1fdaf94cca26ef0"
server_key_version: "encryptionkey@0"
I20260924 10:37:29.083012 16462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 10:37:29.085515 16477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:29.086820 16462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:37:29.086971 16462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "bb1d737edc304e358afb4f04aa620596"
format_stamp: "Formatted at 2026-09-24 10:37:29 on dist-test-slave-8wc7"
server_key: "e05e872bfd5b576ed117ab841a00786f"
server_key_iv: "f5c11af4271f77a2c1fdaf94cca26ef0"
server_key_version: "encryptionkey@0"
I20260924 10:37:29.087103 16462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:29.115332 16462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:29.118904 16462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:29.119139 16462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:29.126670 16462 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:43821
I20260924 10:37:29.126717 16529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:43821 every 8 connection(s)
I20260924 10:37:29.127763 16462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:29.129647 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16462
I20260924 10:37:29.129735 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:37:29.130026 2822 external_mini_cluster.cc:1428] Setting key ca74ad01d7717d44fb3d81ae302a5245
I20260924 10:37:29.130999 16530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:29.137677 16530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596: Bootstrap starting.
Sep 24 10:37:29 dist-test-slave-8wc7 krb5kdc[16449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:29.140389 16530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:29.141346 16530 log.cc:824] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:29.143708 16530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596: No bootstrap required, opened a new log
I20260924 10:37:29.144593 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:29.131450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.2.193.190:43821)
0924 10:37:29.131865 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:29.131881 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:29.131917 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:37:29.132557 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:29.132573 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:29.132898 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:29.133249 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:29.134190 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:29.135190 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:29.136398 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:29.136856 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:29.136966 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:29.139771 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:29.139777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:29.139791 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:29.139821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:29.141780 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:29.142271 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:29.142276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:29.142281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:29.142390 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:29.142681 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:29.142684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:29.142685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:29.143072 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:29.143222 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:29.143529 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 10:37:29.143847 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":139,"threads_started":1}
I20260924 10:37:29.147742 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:37:29.148013 16530 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d737edc304e358afb4f04aa620596" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43821 } }
I20260924 10:37:29.148303 16530 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:29.148377 16530 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1d737edc304e358afb4f04aa620596, State: Initialized, Role: FOLLOWER
I20260924 10:37:29.148835 16530 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d737edc304e358afb4f04aa620596" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43821 } }
I20260924 10:37:29.149005 16530 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:37:29.149075 16530 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:37:29.149201 16530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:29.150242 16530 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d737edc304e358afb4f04aa620596" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43821 } }
I20260924 10:37:29.150556 16530 leader_election.cc:302] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb1d737edc304e358afb4f04aa620596; no voters:
I20260924 10:37:29.150861 16530 leader_election.cc:288] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:37:29.150964 16535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:29.151201 16535 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [term 1 LEADER]: Becoming Leader. State: Replica: bb1d737edc304e358afb4f04aa620596, State: Running, Role: LEADER
I20260924 10:37:29.151567 16535 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d737edc304e358afb4f04aa620596" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43821 } }
I20260924 10:37:29.152175 16530 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:29.153249 16536 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb1d737edc304e358afb4f04aa620596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d737edc304e358afb4f04aa620596" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43821 } } }
I20260924 10:37:29.153386 16536 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:29.153541 16537 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb1d737edc304e358afb4f04aa620596. Latest consensus state: current_term: 1 leader_uuid: "bb1d737edc304e358afb4f04aa620596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d737edc304e358afb4f04aa620596" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 43821 } } }
I20260924 10:37:29.153728 16537 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596 [sys.catalog]: This master's current role is: LEADER
W20260924 10:37:29.157099 16550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:37:29.157253 16550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:37:29.157339 16544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:29.158232 16544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:29.164352 16544 catalog_manager.cc:1347] Generated new cluster ID: dd67b60d267c4b4d90329d1881b296f2
I20260924 10:37:29.164455 16544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:37:29.183904 16544 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:37:29.185285 16544 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:37:29.197522 16544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb1d737edc304e358afb4f04aa620596: Generated new TSK 0
I20260924 10:37:29.198415 16544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:37:29.203634 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16462
2026-09-24T10:37:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1443 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:37:29 dist-test-slave-8wc7 krb5kdc[16557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:37:29 dist-test-slave-8wc7 krb5kdc[16557](info): set up 2 sockets
Sep 24 10:37:29 dist-test-slave-8wc7 krb5kdc[16557](info): commencing operation
krb5kdc: starting...
W20260924 10:37:30.233465 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:37:30 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:37:30.364013 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40973
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:30.478340 16570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:30.478729 16570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:30.478816 16570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:30.483043 16570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:30.483155 16570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:30.483191 16570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:30.483233 16570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:30.483265 16570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:30.488933 16570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:30.490120 16570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:30.491236 16570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:30.500166 16576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:30.500161 16578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:30.500161 16575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:30.500543 16570 server_base.cc:1034] running on GCE node
I20260924 10:37:30.501155 16570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:30.502336 16570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:30.503545 16570 hybrid_clock.cc:648] HybridClock initialized: now 1790246250503516 us; error 47 us; skew 500 ppm
Sep 24 10:37:30 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:30.511258 16570 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:30.512282 16570 webserver.cc:460] Webserver started at http://127.2.193.190:46649/ using document root <none> and password file <none>
I20260924 10:37:30.512897 16570 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:30.512991 16570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:30.513299 16570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:30.515470 16570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "abb8de7263974a189f65a19f9121fe63"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "2253e5e248a154434e89f68844aba554"
server_key_iv: "0562b6afd5e3cd4281185933da0a6299"
server_key_version: "encryptionkey@0"
I20260924 10:37:30.516067 16570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "abb8de7263974a189f65a19f9121fe63"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "2253e5e248a154434e89f68844aba554"
server_key_iv: "0562b6afd5e3cd4281185933da0a6299"
server_key_version: "encryptionkey@0"
I20260924 10:37:30.519991 16570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:37:30.522603 16585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:30.523924 16570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:37:30.524093 16570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "abb8de7263974a189f65a19f9121fe63"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "2253e5e248a154434e89f68844aba554"
server_key_iv: "0562b6afd5e3cd4281185933da0a6299"
server_key_version: "encryptionkey@0"
I20260924 10:37:30.524226 16570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:30.552584 16570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:30.555825 16570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:30.556073 16570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:30.563520 16570 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:40973
I20260924 10:37:30.563823 16637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:40973 every 8 connection(s)
I20260924 10:37:30.564610 16570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:30.567628 16638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:30.571161 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16570
I20260924 10:37:30.571319 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:37:30.571620 2822 external_mini_cluster.cc:1428] Setting key 0879cfc8628b7e6964a3dca26e818f7e
I20260924 10:37:30.574028 16638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Bootstrap starting.
I20260924 10:37:30.576825 16638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:30.577750 16638 log.cc:824] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:30.580050 16638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: No bootstrap required, opened a new log
Sep 24 10:37:30 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:30.583940 16638 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:30.584232 16638 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:30.584303 16638 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb8de7263974a189f65a19f9121fe63, State: Initialized, Role: FOLLOWER
I20260924 10:37:30.584826 16638 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:30.584966 16638 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:37:30.585055 16638 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:37:30.585157 16638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:30.586412 16638 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:30.586782 16638 leader_election.cc:302] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb8de7263974a189f65a19f9121fe63; no voters:
I20260924 10:37:30.587132 16638 leader_election.cc:288] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:37:30.587242 16643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:30.587522 16643 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 LEADER]: Becoming Leader. State: Replica: abb8de7263974a189f65a19f9121fe63, State: Running, Role: LEADER
I20260924 10:37:30.587891 16643 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:30.587985 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:30.573145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.2.193.190:40973)
0924 10:37:30.573648 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:30.573668 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:30.573713 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:37:30.574547 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:30.574572 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:30.574992 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:30.575388 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:30.576034 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:30.577558 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:30.578735 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:30.579393 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:30.579555 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:30.582082 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:30.582088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:30.582105 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:30.582136 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:30.584693 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:30.585231 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:30.585239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:30.585244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:30.585388 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:30.585654 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:30.585658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:30.585659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:30.586176 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:30.586371 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:30.586597 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 10:37:30.586903 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":124,"threads_started":1}
I20260924 10:37:30.588449 16638 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:30.590956 16644 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb8de7263974a189f65a19f9121fe63. Latest consensus state: current_term: 1 leader_uuid: "abb8de7263974a189f65a19f9121fe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } } }
I20260924 10:37:30.591384 16644 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:30.591173 16645 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb8de7263974a189f65a19f9121fe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } } }
I20260924 10:37:30.591698 16645 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: This master's current role is: LEADER
W20260924 10:37:30.593446 16657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:37:30.593533 16657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:37:30.593597 16658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:30.594251 16658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:30.599534 16658 catalog_manager.cc:1347] Generated new cluster ID: 7a359b5fd67a4b368956640c2024ce64
I20260924 10:37:30.599596 16658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:37:30.644979 16658 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:37:30.646041 16658 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:37:30.654944 16658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Generated new TSK 0
I20260924 10:37:30.655720 16658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:37:30.692919 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40973
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:30.810021 16664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:30.810297 16664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:30.810346 16664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:30.814499 16664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:30.814570 16664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:30.814635 16664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:37:30.820222 16664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:30.821422 16664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:30.822562 16664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:30.832581 16672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:30.832732 16669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:30.832746 16670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:30.832854 16664 server_base.cc:1034] running on GCE node
I20260924 10:37:30.833458 16664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:30.834194 16664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:30.835393 16664 hybrid_clock.cc:648] HybridClock initialized: now 1790246250835354 us; error 59 us; skew 500 ppm
Sep 24 10:37:30 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:30.838379 16664 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:37:30.839432 16664 webserver.cc:460] Webserver started at http://127.2.193.129:38911/ using document root <none> and password file <none>
I20260924 10:37:30.840040 16664 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:30.840137 16664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:30.840384 16664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:30.842571 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "98ccb7f73e2c42dc910ea43f285527e1"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "684321b61981c34e59bfda4ff1788a7e"
server_key_iv: "d080cfab4feafadf013f1208d66af08b"
server_key_version: "encryptionkey@0"
I20260924 10:37:30.843159 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "98ccb7f73e2c42dc910ea43f285527e1"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "684321b61981c34e59bfda4ff1788a7e"
server_key_iv: "d080cfab4feafadf013f1208d66af08b"
server_key_version: "encryptionkey@0"
I20260924 10:37:30.846992 16664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:37:30.849429 16679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:30.850641 16664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:37:30.850783 16664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "98ccb7f73e2c42dc910ea43f285527e1"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "684321b61981c34e59bfda4ff1788a7e"
server_key_iv: "d080cfab4feafadf013f1208d66af08b"
server_key_version: "encryptionkey@0"
I20260924 10:37:30.850903 16664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:30.872706 16664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:30.876114 16664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:30.876377 16664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:30.877070 16664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:30.878023 16664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:30.878109 16664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:30.878186 16664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:30.878239 16664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:30.887431 16664 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:42261
I20260924 10:37:30.887451 16792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:42261 every 8 connection(s)
I20260924 10:37:30.888410 16664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:37:30.890182 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16664
I20260924 10:37:30.890318 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:37:30.890609 2822 external_mini_cluster.cc:1428] Setting key 42690b9c33abe9647395f065db52a054
Sep 24 10:37:30 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:37:30.903796 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:30.891961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:36793 (local address 127.2.193.190:40973)
0924 10:37:30.892106 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:30.892111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:30.892127 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:37:30.893000 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:30.893006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:30.893077 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:30.893207 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:30.894466 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:30.895004 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:30.896838 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:30.897081 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:30.897221 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:30.899878 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:30.899884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:30.899889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:30.899923 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:30.901628 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:30.902097 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:30.902100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:30.902101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:30.902222 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:30.902598 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:30.902602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:30.902604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:30.902796 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:30.902905 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:30.903495 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 10:37:30.903630 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:37:30.904731 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:30.890944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:36793)
0924 10:37:30.891449 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:30.891505 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:30.892765 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:30.893410 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:30.893418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:30.893789 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:30.894299 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:30.894312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:30.895145 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:30.895152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:30.896690 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:30.896707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:30.897275 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:30.897279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:30.897515 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:30.898209 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:30.898226 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:30.899691 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:30.901734 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:30.901739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:30.901752 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:30.902008 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:30.902323 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:30.902327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:30.902329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:30.902485 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:30.902916 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:30.902924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:30.903238 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 10:37:30.903442 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:37:30.903868 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260924 10:37:30.906231 16793 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40973
I20260924 10:37:30.906499 16793 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:30.907001 16793 heartbeater.cc:511] Master 127.2.193.190:40973 requested a full tablet report, sending...
I20260924 10:37:30.908529 16602 ts_manager.cc:194] Registered new tserver with Master: 98ccb7f73e2c42dc910ea43f285527e1 (127.2.193.129:42261)
I20260924 10:37:30.909751 16602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:36793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:37:30.928342 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40973
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:31.040529 16798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:31.040807 16798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:31.040882 16798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:31.045073 16798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:31.045145 16798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:31.045303 16798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:37:31.050899 16798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:31.052055 16798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:31.053153 16798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:31.063482 16803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:31.063477 16804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:31.063542 16798 server_base.cc:1034] running on GCE node
W20260924 10:37:31.063477 16806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:31.064123 16798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:31.064756 16798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:31.065913 16798 hybrid_clock.cc:648] HybridClock initialized: now 1790246251065878 us; error 49 us; skew 500 ppm
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:31.068751 16798 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:37:31.069762 16798 webserver.cc:460] Webserver started at http://127.2.193.130:43561/ using document root <none> and password file <none>
I20260924 10:37:31.070348 16798 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:31.070413 16798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:31.070688 16798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:31.072849 16798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "f89b4cc25f9a4514933adfa2204c1a21"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "bd5316cd9af72bbfaee9a8afc985834d"
server_key_iv: "9d494a9b31093d15085f395ee7b99985"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.073464 16798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "f89b4cc25f9a4514933adfa2204c1a21"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "bd5316cd9af72bbfaee9a8afc985834d"
server_key_iv: "9d494a9b31093d15085f395ee7b99985"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.077524 16798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:37:31.080180 16813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.081604 16798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:37:31.081759 16798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "f89b4cc25f9a4514933adfa2204c1a21"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "bd5316cd9af72bbfaee9a8afc985834d"
server_key_iv: "9d494a9b31093d15085f395ee7b99985"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.081889 16798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:31.102080 16798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:31.105137 16798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:31.105427 16798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:31.106067 16798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:31.106922 16798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:31.106997 16798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.107061 16798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:31.107105 16798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.115734 16798 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:40005
I20260924 10:37:31.115798 16926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:40005 every 8 connection(s)
I20260924 10:37:31.116746 16798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:37:31.125308 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16798
I20260924 10:37:31.125440 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:37:31.125734 2822 external_mini_cluster.cc:1428] Setting key 97793ce7b0dd019584c38285e3afa967
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:31.131642 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.118709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:47141 (local address 127.2.193.190:40973)
0924 10:37:31.118847 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.118850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.119738 (+ 888us) server_negotiation.cc:407] Connection header received
0924 10:37:31.120978 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.120981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.121030 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.121109 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:31.122373 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.122984 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.124110 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.124327 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.124406 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.127358 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:31.127362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:31.127364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:31.127395 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:31.129321 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.130037 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.130040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.130041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.130116 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.130527 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.130529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.130530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.130710 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:31.130786 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.131340 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.131479 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:37:31.132525 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.119135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:47141)
0924 10:37:31.119577 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.119622 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.120775 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.121350 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.121358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.121728 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:31.122234 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.122248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.123098 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.123101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.124006 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.124015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.124459 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.124461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.124612 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.125206 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:31.125231 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:31.127135 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:31.129497 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.129507 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.129528 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.129935 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.130209 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.130212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.130216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.130421 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.130805 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:31.130811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:31.131094 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.131295 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.131655 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260924 10:37:31.133765 16927 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40973
I20260924 10:37:31.134048 16927 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:31.134558 16927 heartbeater.cc:511] Master 127.2.193.190:40973 requested a full tablet report, sending...
I20260924 10:37:31.135701 16602 ts_manager.cc:194] Registered new tserver with Master: f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:37:31.136549 16602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:47141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:37:31.156958 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40973
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:31.276528 16932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:31.276962 16932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:31.277047 16932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:31.284035 16932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:31.284121 16932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:31.284224 16932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:37:31.293704 16932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:31.295101 16932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:31.296223 16932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:31.308103 16937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:31.308180 16940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:31.308106 16938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:31.310559 16932 server_base.cc:1034] running on GCE node
I20260924 10:37:31.311035 16932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:31.311770 16932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:31.312949 16932 hybrid_clock.cc:648] HybridClock initialized: now 1790246251312917 us; error 56 us; skew 500 ppm
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:31.316049 16932 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:37:31.317086 16932 webserver.cc:460] Webserver started at http://127.2.193.131:37315/ using document root <none> and password file <none>
I20260924 10:37:31.317762 16932 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:31.317863 16932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:31.318087 16932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:31.320245 16932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "48ee8b456f8f4db3a13554a7146f9c65"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "beaa1b2361c85944d4d8ae110c0b775c"
server_key_iv: "ab0d0f8ee4ab70e8c75ba9c6c5384da1"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.320851 16932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "48ee8b456f8f4db3a13554a7146f9c65"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "beaa1b2361c85944d4d8ae110c0b775c"
server_key_iv: "ab0d0f8ee4ab70e8c75ba9c6c5384da1"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.324920 16932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 10:37:31.327615 16947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.328969 16932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:37:31.329113 16932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "48ee8b456f8f4db3a13554a7146f9c65"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "beaa1b2361c85944d4d8ae110c0b775c"
server_key_iv: "ab0d0f8ee4ab70e8c75ba9c6c5384da1"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.329281 16932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:31.347254 16932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:31.350492 16932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:31.350739 16932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:31.351369 16932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:31.352248 16932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:31.352324 16932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.352401 16932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:31.352445 16932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.362066 16932 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:36647
I20260924 10:37:31.362087 17060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:36647 every 8 connection(s)
I20260924 10:37:31.363050 16932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:37:31.364059 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 16932
I20260924 10:37:31.364187 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:37:31.364460 2822 external_mini_cluster.cc:1428] Setting key 948031094be2736efef2843b26215d76
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:31.376948 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.365036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:48937 (local address 127.2.193.190:40973)
0924 10:37:31.365165 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.365190 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.366306 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 10:37:31.367743 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.367748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.367806 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.367896 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:31.369143 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.369734 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.370877 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.371097 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.371178 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.373810 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:31.373813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:31.373815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:31.373840 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:31.375097 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.375580 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.375583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.375584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.375660 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.375957 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.375959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.375960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.376136 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:31.376212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.376689 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.376794 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 10:37:31.377882 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.365583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:48937)
0924 10:37:31.366138 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.366195 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.367534 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.368101 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.368110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.368517 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:31.369010 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.369024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.369846 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.369848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.370767 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.370776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.371232 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.371235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.371384 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.372037 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:31.372055 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:31.373603 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:31.375206 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.375211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.375225 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.375492 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.375737 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.375739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.375741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.375881 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.376221 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:31.376226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:31.376464 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.376645 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.376933 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260924 10:37:31.379038 17061 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40973
I20260924 10:37:31.379269 17061 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:31.379714 17061 heartbeater.cc:511] Master 127.2.193.190:40973 requested a full tablet report, sending...
I20260924 10:37:31.380733 16601 ts_manager.cc:194] Registered new tserver with Master: 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647)
I20260924 10:37:31.381412 16601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:48937
I20260924 10:37:31.390780 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:37:31.400033 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.392833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.2.193.190:40973)
0924 10:37:31.392969 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.392972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.393154 (+ 182us) server_negotiation.cc:407] Connection header received
0924 10:37:31.393333 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.393335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.393379 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.393451 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:31.394170 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.394692 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.395489 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.395704 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.395781 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.397122 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:31.397125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:31.397127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:31.397151 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:31.398630 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.398986 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.398989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.398990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.399062 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.399281 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.399283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.399284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.399502 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:31.399619 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.399810 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.399909 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:37:31.403151 16601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:37:31.406414 16601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:37:31.421366 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.415963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:42261 (local address 127.0.0.1:52764)
0924 10:37:31.416283 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.416296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.416369 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.417080 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.417085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.417105 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:31.417229 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.417235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.418580 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.418584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.419742 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.419751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.420772 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.420774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.420856 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.420888 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.421045 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.421207 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260924 10:37:31.421375 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.416130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.2.193.129:42261)
0924 10:37:31.416605 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.416612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.416634 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:37:31.416715 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.416722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.416907 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.417110 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:31.417366 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.418472 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.419872 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.420652 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.420772 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.420930 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.421083 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.421229 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":86,"threads_started":1}
I20260924 10:37:31.421615 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.415143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:40005 (local address 127.0.0.1:55994)
0924 10:37:31.415991 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.416020 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.416111 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.417061 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.417065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.417083 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:31.417328 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.417337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.418308 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.418312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.420034 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.420046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.421133 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.421137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.421277 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.421302 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.421353 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.421422 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":849,"spinlock_wait_cycles":3840,"thread_start_us":87,"threads_started":1}
I20260924 10:37:31.422291 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.415298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.2.193.130:40005)
0924 10:37:31.416666 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.416677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.416696 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:37:31.416759 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.416762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.416906 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.417248 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:31.417423 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.418159 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.420166 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.420995 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.421819 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.421971 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.422027 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.422128 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":133,"threads_started":1}
I20260924 10:37:31.423627 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.415146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:36647 (local address 127.0.0.1:47828)
0924 10:37:31.415568 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.415624 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.415754 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.416784 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.416789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.416808 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:31.416912 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.416916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.418423 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.418426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.420035 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.420046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.423251 (+ 3205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.423254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.423346 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.423368 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.423431 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.423503 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":153,"thread_start_us":108,"threads_started":1}
I20260924 10:37:31.424121 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.415413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.2.193.131:36647)
0924 10:37:31.416032 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.416047 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.416072 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:37:31.416150 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.416156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.416386 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.416578 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:31.417036 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.418284 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.421023 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.422159 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.422262 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.423832 (+ 1570us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.423887 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.423976 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":155,"threads_started":1}
I20260924 10:37:31.424820 16727 tablet_service.cc:1467] Processing CreateTablet for tablet e61964fa5c724125bcf6e433af06e4f3 (DEFAULT_TABLE table=test-table [id=e3047ea32f264f419c0814598eedead6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:31.424872 16861 tablet_service.cc:1467] Processing CreateTablet for tablet e61964fa5c724125bcf6e433af06e4f3 (DEFAULT_TABLE table=test-table [id=e3047ea32f264f419c0814598eedead6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:31.425892 16727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61964fa5c724125bcf6e433af06e4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:31.425870 16861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61964fa5c724125bcf6e433af06e4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:31.428097 16995 tablet_service.cc:1467] Processing CreateTablet for tablet e61964fa5c724125bcf6e433af06e4f3 (DEFAULT_TABLE table=test-table [id=e3047ea32f264f419c0814598eedead6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:31.429219 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61964fa5c724125bcf6e433af06e4f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:31.432883 17078 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Bootstrap starting.
I20260924 10:37:31.434242 17079 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap starting.
I20260924 10:37:31.435931 17078 tablet_bootstrap.cc:654] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:31.435956 17080 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Bootstrap starting.
I20260924 10:37:31.436466 17079 tablet_bootstrap.cc:654] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:31.437250 17078 log.cc:824] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:31.437520 17079 log.cc:824] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:31.438802 17080 tablet_bootstrap.cc:654] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:31.439401 17078 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: No bootstrap required, opened a new log
I20260924 10:37:31.439580 17079 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: No bootstrap required, opened a new log
I20260924 10:37:31.439621 17078 ts_tablet_manager.cc:1397] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:37:31.439666 17080 log.cc:824] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:31.439772 17079 ts_tablet_manager.cc:1397] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:37:31.441823 17080 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: No bootstrap required, opened a new log
I20260924 10:37:31.442013 17080 ts_tablet_manager.cc:1397] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:37:31.442962 17078 raft_consensus.cc:348] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.443243 17078 raft_consensus.cc:374] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:31.443300 17078 raft_consensus.cc:729] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89b4cc25f9a4514933adfa2204c1a21, State: Initialized, Role: FOLLOWER
I20260924 10:37:31.443815 17078 consensus_queue.cc:260] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.444345 17079 raft_consensus.cc:348] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.444610 17079 raft_consensus.cc:374] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:31.444671 17079 raft_consensus.cc:729] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ccb7f73e2c42dc910ea43f285527e1, State: Initialized, Role: FOLLOWER
I20260924 10:37:31.444943 17078 ts_tablet_manager.cc:1428] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260924 10:37:31.445120 17079 consensus_queue.cc:260] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.445363 16927 heartbeater.cc:503] Master 127.2.193.190:40973 was elected leader, sending a full tablet report...
I20260924 10:37:31.445781 16793 heartbeater.cc:503] Master 127.2.193.190:40973 was elected leader, sending a full tablet report...
I20260924 10:37:31.445849 17080 raft_consensus.cc:348] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.446089 17080 raft_consensus.cc:374] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:31.446107 17079 ts_tablet_manager.cc:1428] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:37:31.446142 17080 raft_consensus.cc:729] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ee8b456f8f4db3a13554a7146f9c65, State: Initialized, Role: FOLLOWER
I20260924 10:37:31.446522 17080 consensus_queue.cc:260] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.447186 17061 heartbeater.cc:503] Master 127.2.193.190:40973 was elected leader, sending a full tablet report...
I20260924 10:37:31.447427 17080 ts_tablet_manager.cc:1428] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260924 10:37:31.468577 17085 raft_consensus.cc:482] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:31.468848 17085 raft_consensus.cc:504] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.470137 17085 leader_election.cc:288] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005), 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647)
I20260924 10:37:31.474455 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.470412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:33575 (local address 127.2.193.130:40005)
0924 10:37:31.470581 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.470588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.470934 (+ 346us) server_negotiation.cc:407] Connection header received
0924 10:37:31.471086 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.471092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.471151 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.471213 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:31.471567 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.472164 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.473391 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.473807 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.473937 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.473995 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.474231 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.474316 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:37:31.474455 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.470578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:51519 (local address 127.2.193.131:36647)
0924 10:37:31.470712 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.470719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.470978 (+ 259us) server_negotiation.cc:407] Connection header received
0924 10:37:31.471139 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.471142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.471182 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.471278 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:31.471657 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.472245 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.473427 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.473829 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.473965 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.474020 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.474234 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.474314 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:37:31.474488 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.470329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:40005 (local address 127.2.193.129:33575)
0924 10:37:31.470822 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.470834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.470960 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.471245 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.471252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.471272 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:31.471451 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.471458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.472287 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.472293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.473272 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.473282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.473936 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.473941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.474050 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.474122 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.474213 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.474320 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":83,"thread_start_us":130,"threads_started":1}
I20260924 10:37:31.474490 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.470460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:36647 (local address 127.2.193.129:51519)
0924 10:37:31.470852 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.470863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.471029 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.471313 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.471318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.471334 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:31.471485 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.471521 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.472361 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.472363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.473287 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.473296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.473984 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.473988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.474104 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.474124 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.474206 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.474326 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":142,"thread_start_us":115,"threads_started":1}
I20260924 10:37:31.475402 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61964fa5c724125bcf6e433af06e4f3" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ee8b456f8f4db3a13554a7146f9c65" is_pre_election: true
I20260924 10:37:31.475410 16881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61964fa5c724125bcf6e433af06e4f3" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89b4cc25f9a4514933adfa2204c1a21" is_pre_election: true
I20260924 10:37:31.475777 17015 raft_consensus.cc:2454] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 0.
I20260924 10:37:31.475772 16881 raft_consensus.cc:2454] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 0.
I20260924 10:37:31.476279 16682 leader_election.cc:302] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48ee8b456f8f4db3a13554a7146f9c65, 98ccb7f73e2c42dc910ea43f285527e1, f89b4cc25f9a4514933adfa2204c1a21; no voters:
I20260924 10:37:31.476583 17085 raft_consensus.cc:2781] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:37:31.476668 17085 raft_consensus.cc:482] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:37:31.476713 17085 raft_consensus.cc:3037] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:31.477807 17085 raft_consensus.cc:504] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.478188 17085 leader_election.cc:288] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 election: Requested vote from peers f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005), 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647)
I20260924 10:37:31.478567 16881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61964fa5c724125bcf6e433af06e4f3" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89b4cc25f9a4514933adfa2204c1a21"
I20260924 10:37:31.478590 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61964fa5c724125bcf6e433af06e4f3" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ee8b456f8f4db3a13554a7146f9c65"
I20260924 10:37:31.478688 16881 raft_consensus.cc:3037] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:31.478718 17015 raft_consensus.cc:3037] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:31.479768 17015 raft_consensus.cc:2454] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 1.
I20260924 10:37:31.479769 16881 raft_consensus.cc:2454] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 1.
I20260924 10:37:31.480083 16681 leader_election.cc:302] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ee8b456f8f4db3a13554a7146f9c65, 98ccb7f73e2c42dc910ea43f285527e1; no voters:
I20260924 10:37:31.480291 17085 raft_consensus.cc:2781] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:31.480541 17085 raft_consensus.cc:686] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 LEADER]: Becoming Leader. State: Replica: 98ccb7f73e2c42dc910ea43f285527e1, State: Running, Role: LEADER
I20260924 10:37:31.480896 17085 consensus_queue.cc:237] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.484318 16602 catalog_manager.cc:5495] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98ccb7f73e2c42dc910ea43f285527e1 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "98ccb7f73e2c42dc910ea43f285527e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } health_report { overall_health: HEALTHY } } }
I20260924 10:37:31.499540 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.496115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.2.193.129:42261)
0924 10:37:31.496254 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.496257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.496271 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:37:31.496418 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.496420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.496473 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.496564 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:37:31.496805 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.497383 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.498267 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.498470 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.498546 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.498803 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:37:31.499235 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:37:31.499326 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.499380 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.499431 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:37:31.505636 17015 raft_consensus.cc:1261] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Refusing update from remote peer 98ccb7f73e2c42dc910ea43f285527e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:31.505635 16881 raft_consensus.cc:1261] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Refusing update from remote peer 98ccb7f73e2c42dc910ea43f285527e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:31.506420 17085 consensus_queue.cc:1035] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:31.506631 17089 consensus_queue.cc:1035] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:31.525451 16600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:31.525625 16600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:31.528119 16600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Sending DeleteTablet for 3 replicas of tablet e61964fa5c724125bcf6e433af06e4f3
I20260924 10:37:31.528970 16861 tablet_service.cc:1514] Processing DeleteTablet for tablet e61964fa5c724125bcf6e433af06e4f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:31 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:55994
I20260924 10:37:31.529127 16995 tablet_service.cc:1514] Processing DeleteTablet for tablet e61964fa5c724125bcf6e433af06e4f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:31 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:47828
I20260924 10:37:31.529248 16727 tablet_service.cc:1514] Processing DeleteTablet for tablet e61964fa5c724125bcf6e433af06e4f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:31 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52764
I20260924 10:37:31.529392 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16664
I20260924 10:37:31.529564 17102 tablet_replica.cc:330] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: stopping tablet replica
I20260924 10:37:31.529903 17102 raft_consensus.cc:2229] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:31.530241 17102 raft_consensus.cc:2258] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:31.530267 17103 tablet_replica.cc:330] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: stopping tablet replica
I20260924 10:37:31.530661 17103 raft_consensus.cc:2229] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:31.531018 17103 raft_consensus.cc:2258] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:31.532164 17102 ts_tablet_manager.cc:1905] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:31.532598 17103 ts_tablet_manager.cc:1905] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:31.534796 17102 ts_tablet_manager.cc:1918] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:31.534915 17102 log.cc:1196] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/e61964fa5c724125bcf6e433af06e4f3
I20260924 10:37:31.535248 17102 ts_tablet_manager.cc:1939] T e61964fa5c724125bcf6e433af06e4f3 P f89b4cc25f9a4514933adfa2204c1a21: Deleting consensus metadata
I20260924 10:37:31.535974 17103 ts_tablet_manager.cc:1918] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:31.536080 17103 log.cc:1196] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/e61964fa5c724125bcf6e433af06e4f3
I20260924 10:37:31.536415 17103 ts_tablet_manager.cc:1939] T e61964fa5c724125bcf6e433af06e4f3 P 48ee8b456f8f4db3a13554a7146f9c65: Deleting consensus metadata
I20260924 10:37:31.536514 16588 catalog_manager.cc:4841] TS f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005): tablet e61964fa5c724125bcf6e433af06e4f3 (table test-table [id=e3047ea32f264f419c0814598eedead6]) successfully deleted
W20260924 10:37:31.537595 16587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:42261 (error 108)
I20260924 10:37:31.537845 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16798
I20260924 10:37:31.538096 16587 catalog_manager.cc:4841] TS 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647): tablet e61964fa5c724125bcf6e433af06e4f3 (table test-table [id=e3047ea32f264f419c0814598eedead6]) successfully deleted
I20260924 10:37:31.541749 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.538946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:42261 (local address 127.0.0.1:52794)
0924 10:37:31.541546 (+ 2600us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.541638 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:42261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2477}
W20260924 10:37:31.541956 16587 catalog_manager.cc:4584] TS 98ccb7f73e2c42dc910ea43f285527e1 (127.2.193.129:42261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e61964fa5c724125bcf6e433af06e4f3: Network error: Client connection negotiation failed: client connection to 127.2.193.129:42261: connect: Connection refused (error 111)
I20260924 10:37:31.543615 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16932
I20260924 10:37:31.549269 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 16570
I20260924 10:37:31.555537 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40973
--webserver_interface=127.2.193.190
--webserver_port=46649
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:31.669373 17105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:31.669685 17105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:31.669729 17105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:31.674019 17105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:31.674093 17105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:31.674113 17105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:31.674130 17105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:31.674146 17105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:31.679867 17105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=46649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:31.681071 17105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:31.682230 17105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:31.696024 17111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:31.696024 17112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:31.696084 17114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:31.696318 17105 server_base.cc:1034] running on GCE node
I20260924 10:37:31.697055 17105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:31.698309 17105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:31.699476 17105 hybrid_clock.cc:648] HybridClock initialized: now 1790246251699453 us; error 37 us; skew 500 ppm
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:31.702419 17105 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:31.703393 17105 webserver.cc:460] Webserver started at http://127.2.193.190:46649/ using document root <none> and password file <none>
I20260924 10:37:31.703958 17105 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:31.704033 17105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:31.708038 17105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:37:31.710346 17121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.711674 17105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:37:31.711830 17105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "abb8de7263974a189f65a19f9121fe63"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "2253e5e248a154434e89f68844aba554"
server_key_iv: "0562b6afd5e3cd4281185933da0a6299"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.712476 17105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:31.726886 17105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:31.730093 17105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:31.730319 17105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:31.738835 17105 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:40973
I20260924 10:37:31.739073 17173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:40973 every 8 connection(s)
I20260924 10:37:31.740080 17105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:31.741894 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 17105
I20260924 10:37:31.742364 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:42261
--local_ip_for_outbound_sockets=127.2.193.129
--tserver_master_addrs=127.2.193.190:40973
--webserver_port=38911
--webserver_interface=127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:37:31.745507 17174 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:37:31.747367 17174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Bootstrap starting.
I20260924 10:37:31.764930 17174 log.cc:824] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:31.773895 17174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:37:31.774318 17174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Bootstrap complete.
I20260924 10:37:31.778614 17174 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:31.779028 17174 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb8de7263974a189f65a19f9121fe63, State: Initialized, Role: FOLLOWER
I20260924 10:37:31.779579 17174 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:31.779702 17174 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:37:31.779795 17174 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:37:31.779911 17174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:37:31.781984 17174 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:31.782334 17174 leader_election.cc:302] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb8de7263974a189f65a19f9121fe63; no voters:
I20260924 10:37:31.782680 17174 leader_election.cc:288] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:37:31.782765 17178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:37:31.782964 17178 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 LEADER]: Becoming Leader. State: Replica: abb8de7263974a189f65a19f9121fe63, State: Running, Role: LEADER
I20260924 10:37:31.783260 17178 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:31.783962 17174 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:31.784293 17180 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb8de7263974a189f65a19f9121fe63. Latest consensus state: current_term: 2 leader_uuid: "abb8de7263974a189f65a19f9121fe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } } }
I20260924 10:37:31.784430 17180 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:31.784643 17179 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "abb8de7263974a189f65a19f9121fe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } } }
I20260924 10:37:31.784756 17179 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:31.784801 17182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:31.789918 17182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:31.790587 17182 catalog_manager.cc:1259] Loaded cluster ID: 7a359b5fd67a4b368956640c2024ce64
I20260924 10:37:31.790655 17182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:37:31.792074 17182 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 10:37:31.796972 17182 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 10:37:31.802865 17182 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:37:31.803686 17182 catalog_manager.cc:5879] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Loaded TSK: 0
I20260924 10:37:31.804925 17182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:37:31.861720 17176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:31.862053 17176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:31.862131 17176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:31.866451 17176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:31.866559 17176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:31.866653 17176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:37:31.872404 17176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:42261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=38911
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:31.873656 17176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:31.874713 17176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:31.885914 17201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:31.885972 17202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:31.885914 17204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:31.886345 17176 server_base.cc:1034] running on GCE node
I20260924 10:37:31.886803 17176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:31.887585 17176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:31.888775 17176 hybrid_clock.cc:648] HybridClock initialized: now 1790246251888742 us; error 50 us; skew 500 ppm
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:31.891981 17176 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:37:31.893067 17176 webserver.cc:460] Webserver started at http://127.2.193.129:38911/ using document root <none> and password file <none>
I20260924 10:37:31.893690 17176 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:31.893769 17176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:31.898038 17176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:37:31.900475 17211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:31.902055 17176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:37:31.902215 17176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "98ccb7f73e2c42dc910ea43f285527e1"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "684321b61981c34e59bfda4ff1788a7e"
server_key_iv: "d080cfab4feafadf013f1208d66af08b"
server_key_version: "encryptionkey@0"
I20260924 10:37:31.902842 17176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:31.916638 17176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:31.920131 17176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:31.920352 17176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:31.920990 17176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:31.922304 17218 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:37:31.926577 17176 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:37:31.926652 17176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 10:37:31.926702 17176 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:37:31.928509 17176 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:37:31.928570 17176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:37:31.928843 17218 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap starting.
I20260924 10:37:31.939805 17176 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:42261
I20260924 10:37:31.939841 17325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:42261 every 8 connection(s)
I20260924 10:37:31.941244 17176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:37:31.949859 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 17176
I20260924 10:37:31.950335 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:40005
--local_ip_for_outbound_sockets=127.2.193.130
--tserver_master_addrs=127.2.193.190:40973
--webserver_port=43561
--webserver_interface=127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:37:31.950716 17218 log.cc:824] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:31.959600 17218 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:37:31.960057 17218 tablet_bootstrap.cc:492] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap complete.
I20260924 10:37:31.961489 17218 ts_tablet_manager.cc:1397] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.018s
I20260924 10:37:31.962164 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.943580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:37531)
0924 10:37:31.944562 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.944622 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.945967 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.947233 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.947242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.947768 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:31.948571 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.948595 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.950220 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.950224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.951470 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.951480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.952617 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.952621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.952812 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.953497 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:31.953520 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:31.955593 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:31.958305 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.958314 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.958333 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.958741 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.959183 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.959190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.959195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.959399 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.960239 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:31.960245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:31.960540 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.960743 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.961115 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":22,"thread_start_us":126,"threads_started":1}
I20260924 10:37:31.962762 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.943590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:37531 (local address 127.2.193.190:40973)
0924 10:37:31.944231 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.944256 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.944765 (+ 509us) server_negotiation.cc:407] Connection header received
0924 10:37:31.946438 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.946450 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.946779 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.947059 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:31.948764 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.950048 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.951618 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.952446 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.952643 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.955841 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:31.955848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:31.955863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:31.955897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:31.958162 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.958886 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.958892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.958898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.959066 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.959525 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.959528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.959530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.960045 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:31.960185 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.960916 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.961327 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":133,"threads_started":1}
I20260924 10:37:31.964268 17218 raft_consensus.cc:348] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.964805 17218 raft_consensus.cc:729] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ccb7f73e2c42dc910ea43f285527e1, State: Initialized, Role: FOLLOWER
I20260924 10:37:31.965225 17326 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40973
I20260924 10:37:31.965353 17218 consensus_queue.cc:260] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:31.965477 17326 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:31.966037 17326 heartbeater.cc:511] Master 127.2.193.190:40973 requested a full tablet report, sending...
I20260924 10:37:31.966323 17218 ts_tablet_manager.cc:1428] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:37:31.968665 17132 ts_manager.cc:194] Registered new tserver with Master: 98ccb7f73e2c42dc910ea43f285527e1 (127.2.193.129:42261)
I20260924 10:37:31.972611 17132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:37531
I20260924 10:37:31.975575 17326 heartbeater.cc:503] Master 127.2.193.190:40973 was elected leader, sending a full tablet report...
Sep 24 10:37:31 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.129@KRBTEST.COM
I20260924 10:37:31.982317 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.971612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:42261 (local address 127.0.0.1:52802)
0924 10:37:31.972083 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:31.972139 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:31.972274 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:31.972927 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:31.972934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:31.973456 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:31.973598 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.973608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.974610 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.974615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.975328 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:31.975337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.976005 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.976008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:31.976131 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.976768 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:31.976782 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:31.978572 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:31.980416 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.980426 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.980431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.980681 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.980993 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:31.980999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:31.981003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:31.981090 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:31.981627 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:31.981633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:31.981775 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 10:37:31.981912 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:37:31.982125 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":135,"threads_started":1}
I20260924 10:37:31.982321 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:31.971708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.2.193.129:42261)
0924 10:37:31.972417 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:31.972428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:31.972452 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:37:31.972536 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:31.972541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:31.972755 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:31.972900 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:31.973726 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.974487 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.975447 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:31.975869 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:31.975989 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:31.978716 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:31.978725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:31.978737 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:31.978769 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:31.980295 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.980801 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.980807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.980809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.980896 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:31.981243 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:31.981246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:31.981248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:31.981498 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:31.981640 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:31.982011 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 10:37:31.982177 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":86,"threads_started":1}
I20260924 10:37:31.983523 17260 tablet_service.cc:1514] Processing DeleteTablet for tablet e61964fa5c724125bcf6e433af06e4f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:31 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52802
I20260924 10:37:31.984459 17335 tablet_replica.cc:330] stopping tablet replica
I20260924 10:37:31.984786 17335 raft_consensus.cc:2229] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:31.985061 17335 raft_consensus.cc:2258] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:31.986229 17335 ts_tablet_manager.cc:1905] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:31.991125 17335 ts_tablet_manager.cc:1918] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:31.991240 17335 log.cc:1196] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/e61964fa5c724125bcf6e433af06e4f3
I20260924 10:37:31.991559 17335 ts_tablet_manager.cc:1939] T e61964fa5c724125bcf6e433af06e4f3 P 98ccb7f73e2c42dc910ea43f285527e1: Deleting consensus metadata
I20260924 10:37:31.993865 17123 catalog_manager.cc:4841] TS 98ccb7f73e2c42dc910ea43f285527e1 (127.2.193.129:42261): tablet e61964fa5c724125bcf6e433af06e4f3 (table test-table [id=e3047ea32f264f419c0814598eedead6]) successfully deleted
W20260924 10:37:32.075490 17331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:32.075835 17331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:32.075922 17331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:32.080088 17331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:32.080188 17331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:32.080282 17331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:37:32.086202 17331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:40005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=43561
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:32.087357 17331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:32.088456 17331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:32.098332 17340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:32.098331 17343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:32.098344 17341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:32.098531 17331 server_base.cc:1034] running on GCE node
I20260924 10:37:32.099102 17331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:32.099824 17331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:32.101025 17331 hybrid_clock.cc:648] HybridClock initialized: now 1790246252101008 us; error 32 us; skew 500 ppm
Sep 24 10:37:32 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246252, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:32.103775 17331 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:37:32.104740 17331 webserver.cc:460] Webserver started at http://127.2.193.130:43561/ using document root <none> and password file <none>
I20260924 10:37:32.105394 17331 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:32.105511 17331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:32.109633 17331 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:37:32.111837 17350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:32.113092 17331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:37:32.113258 17331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "f89b4cc25f9a4514933adfa2204c1a21"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "bd5316cd9af72bbfaee9a8afc985834d"
server_key_iv: "9d494a9b31093d15085f395ee7b99985"
server_key_version: "encryptionkey@0"
I20260924 10:37:32.113876 17331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:32.143200 17331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:32.146376 17331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:32.146622 17331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:32.147275 17331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:32.148170 17331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:32.148248 17331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:32.148324 17331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:32.148366 17331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:32.157784 17331 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:40005
I20260924 10:37:32.157804 17463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:40005 every 8 connection(s)
I20260924 10:37:32.158934 17331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:37:32.168715 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 17331
I20260924 10:37:32.169239 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:36647
--local_ip_for_outbound_sockets=127.2.193.131
--tserver_master_addrs=127.2.193.190:40973
--webserver_port=37315
--webserver_interface=127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:37:32 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246252, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:32.174376 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.161028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:54357 (local address 127.2.193.190:40973)
0924 10:37:32.161220 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.161225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.162644 (+ 1419us) server_negotiation.cc:407] Connection header received
0924 10:37:32.163613 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.163617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.163683 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.163799 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:32.165069 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.165869 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.167149 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.167404 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.167553 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.170512 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:32.170528 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:32.170531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:32.170559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:32.172123 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:32.172699 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:32.172703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:32.172705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:32.172794 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:32.173161 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:32.173166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:32.173167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:32.173390 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:32.173502 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.174050 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.174201 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:37:32.175298 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.161529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:54357)
0924 10:37:32.162459 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:32.162515 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:32.163378 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:32.164003 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:32.164011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:32.164394 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:32.164903 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.164917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.166055 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.166058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.167004 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.167015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.167599 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.167603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.167792 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.168417 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:32.168444 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:32.170287 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:32.172262 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:32.172269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:32.172288 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:32.172582 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:32.172892 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:32.172895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:32.172897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:32.173060 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:32.173502 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:32.173510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:32.173789 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 10:37:32.173996 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 10:37:32.174395 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":562,"threads_started":1}
I20260924 10:37:32.176644 17464 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40973
I20260924 10:37:32.176956 17464 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:32.177693 17464 heartbeater.cc:511] Master 127.2.193.190:40973 requested a full tablet report, sending...
I20260924 10:37:32.179092 17132 ts_manager.cc:194] Registered new tserver with Master: f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
I20260924 10:37:32.180059 17132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:54357
W20260924 10:37:32.293002 17467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:32.293340 17467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:32.293390 17467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:32.297613 17467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:32.297678 17467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:32.297742 17467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:37:32.303088 17467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:36647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=37315
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:32.304323 17467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:32.305449 17467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:32.315629 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:32.315623 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:32.315740 17467 server_base.cc:1034] running on GCE node
W20260924 10:37:32.315623 17475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:32.316390 17467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:32.317216 17467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:32.318391 17467 hybrid_clock.cc:648] HybridClock initialized: now 1790246252318387 us; error 38 us; skew 500 ppm
Sep 24 10:37:32 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246252, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:32.321218 17467 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:37:32.322163 17467 webserver.cc:460] Webserver started at http://127.2.193.131:37315/ using document root <none> and password file <none>
I20260924 10:37:32.322750 17467 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:32.322851 17467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:32.326958 17467 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 10:37:32.329124 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:32.330467 17467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:37:32.330600 17467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "48ee8b456f8f4db3a13554a7146f9c65"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "beaa1b2361c85944d4d8ae110c0b775c"
server_key_iv: "ab0d0f8ee4ab70e8c75ba9c6c5384da1"
server_key_version: "encryptionkey@0"
I20260924 10:37:32.331156 17467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:32.349612 17467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:32.352662 17467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:32.352872 17467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:32.353574 17467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:32.354429 17467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:32.354475 17467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:32.354552 17467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:32.354593 17467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:32.363909 17467 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:36647
I20260924 10:37:32.363938 17595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:36647 every 8 connection(s)
I20260924 10:37:32.364980 17467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:37:32.366163 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 17467
Sep 24 10:37:32 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246252, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:32.375746 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.366918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.2.193.190:40973)
0924 10:37:32.367227 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.367232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.367386 (+ 154us) server_negotiation.cc:407] Connection header received
0924 10:37:32.367556 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.367559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.367615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.367716 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:32.368532 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.369079 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.370117 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.370366 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.370495 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.371992 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:32.371996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:32.371999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:32.372024 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:32.373870 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:32.374313 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:32.374318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:32.374319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:32.374416 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:32.374704 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:32.374708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:32.374710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:32.374923 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:32.375108 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.375366 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.375481 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":247,"thread_start_us":88,"threads_started":1}
I20260924 10:37:32.379554 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.366914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:49809 (local address 127.2.193.190:40973)
0924 10:37:32.367036 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.367041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.368068 (+ 1027us) server_negotiation.cc:407] Connection header received
0924 10:37:32.369399 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.369402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.369456 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.369540 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:32.370677 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.371407 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.372710 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.372925 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.373034 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.375726 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:32.375729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:32.375732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:32.375758 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:32.377425 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:32.378037 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:32.378041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:32.378042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:32.378126 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:32.378486 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:32.378490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:32.378492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:32.378672 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:32.378777 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.379290 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.379408 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:37:32.380442 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.367425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:49809)
0924 10:37:32.367892 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:32.367944 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:32.369140 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:32.369707 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:32.369714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:32.370072 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:32.370551 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.370563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.371549 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.371552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.372586 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.372596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.373076 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.373080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.373323 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.373857 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:32.373872 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:32.375401 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:32.377546 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:32.377552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:32.377570 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:32.377919 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:32.378218 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:32.378222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:32.378224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:32.378385 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:32.378779 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:32.378787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:32.379056 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 10:37:32.379252 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 10:37:32.379607 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":135,"threads_started":1}
I20260924 10:37:32.381594 17596 heartbeater.cc:347] Connected to a master server at 127.2.193.190:40973
I20260924 10:37:32.381847 17596 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:32.382341 17596 heartbeater.cc:511] Master 127.2.193.190:40973 requested a full tablet report, sending...
I20260924 10:37:32.383395 17132 ts_manager.cc:194] Registered new tserver with Master: 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647)
I20260924 10:37:32.383957 17132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:49809
I20260924 10:37:32.389628 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:37:32.394181 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.390253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.2.193.190:40973)
0924 10:37:32.390378 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.390382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.390625 (+ 243us) server_negotiation.cc:407] Connection header received
0924 10:37:32.390774 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.390776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.390877 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.390963 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:37:32.391193 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.391768 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.392790 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.393003 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.393075 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.393462 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:37:32.393858 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:37:32.393949 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.394026 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.394074 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:37:32.394819 17132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:37:32.398106 17132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:37:32.405928 17260 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f34383f65488b9a1fb0ca0346399b (DEFAULT_TABLE table=test-table [id=2a4d6b480d9a42acbec85714c8907967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:32.406698 17260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f34383f65488b9a1fb0ca0346399b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:32.409936 17218 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap starting.
I20260924 10:37:32.410537 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.404714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:36647 (local address 127.0.0.1:47836)
0924 10:37:32.404820 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:32.404837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:32.404924 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:32.406076 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:32.406079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:32.406098 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:32.406203 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.406210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.407697 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.407701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.408894 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.408905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.410099 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.410103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.410198 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.410250 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 10:37:32.410311 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 10:37:32.410400 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 10:37:32.410909 17218 tablet_bootstrap.cc:654] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:32.411192 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.405022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.2.193.131:36647)
0924 10:37:32.405522 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.405536 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.405560 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:37:32.405643 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.405649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.405912 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.406056 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:32.406344 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.407539 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.409064 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.409959 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.410701 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.410855 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.410909 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.411042 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":134,"threads_started":1}
I20260924 10:37:32.411530 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.405382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.2.193.130:40005)
0924 10:37:32.406379 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.406390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.406410 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:37:32.406472 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.406476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.406693 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.406846 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:32.407274 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.408368 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.409539 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.410416 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.410520 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.410660 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.411319 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.411419 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":106,"threads_started":1}
I20260924 10:37:32.411267 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.405163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:40005 (local address 127.0.0.1:55996)
0924 10:37:32.405483 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:32.405496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:32.405584 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:32.407032 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:32.407035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:32.407055 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:32.407161 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.407166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.408506 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.408512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.409391 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.409405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.410723 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.410727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.410849 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.410900 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 10:37:32.410975 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:37:32.411048 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":103,"threads_started":1}
I20260924 10:37:32.412513 17218 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: No bootstrap required, opened a new log
I20260924 10:37:32.412633 17218 ts_tablet_manager.cc:1397] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 10:37:32.413134 17218 raft_consensus.cc:348] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.413293 17218 raft_consensus.cc:374] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:32.413335 17218 raft_consensus.cc:729] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ccb7f73e2c42dc910ea43f285527e1, State: Initialized, Role: FOLLOWER
I20260924 10:37:32.413494 17218 consensus_queue.cc:260] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.414361 17218 ts_tablet_manager.cc:1428] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:37:32.414714 17530 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f34383f65488b9a1fb0ca0346399b (DEFAULT_TABLE table=test-table [id=2a4d6b480d9a42acbec85714c8907967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:32.414872 17398 tablet_service.cc:1467] Processing CreateTablet for tablet bd9f34383f65488b9a1fb0ca0346399b (DEFAULT_TABLE table=test-table [id=2a4d6b480d9a42acbec85714c8907967]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:32.415936 17398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f34383f65488b9a1fb0ca0346399b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:32.415940 17530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9f34383f65488b9a1fb0ca0346399b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:32.423156 17609 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Bootstrap starting.
I20260924 10:37:32.423202 17610 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Bootstrap starting.
I20260924 10:37:32.425575 17610 tablet_bootstrap.cc:654] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:32.425606 17609 tablet_bootstrap.cc:654] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:32.426406 17610 log.cc:824] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:32.426420 17609 log.cc:824] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:32.428360 17609 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: No bootstrap required, opened a new log
I20260924 10:37:32.428582 17609 ts_tablet_manager.cc:1397] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:37:32.429915 17610 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: No bootstrap required, opened a new log
I20260924 10:37:32.430109 17610 ts_tablet_manager.cc:1397] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:37:32.432667 17609 raft_consensus.cc:348] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.432974 17609 raft_consensus.cc:374] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:32.433034 17609 raft_consensus.cc:729] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89b4cc25f9a4514933adfa2204c1a21, State: Initialized, Role: FOLLOWER
I20260924 10:37:32.433548 17609 consensus_queue.cc:260] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.433889 17610 raft_consensus.cc:348] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.434150 17610 raft_consensus.cc:374] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:32.434211 17610 raft_consensus.cc:729] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ee8b456f8f4db3a13554a7146f9c65, State: Initialized, Role: FOLLOWER
I20260924 10:37:32.434300 17464 heartbeater.cc:503] Master 127.2.193.190:40973 was elected leader, sending a full tablet report...
I20260924 10:37:32.434660 17609 ts_tablet_manager.cc:1428] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:37:32.434649 17610 consensus_queue.cc:260] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.435299 17596 heartbeater.cc:503] Master 127.2.193.190:40973 was elected leader, sending a full tablet report...
I20260924 10:37:32.435591 17610 ts_tablet_manager.cc:1428] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 10:37:32.442646 17327 tablet.cc:2369] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:37:32.616016 17597 tablet.cc:2369] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:37:32.660897 17465 tablet.cc:2369] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:37:32.792227 17332 raft_consensus.cc:482] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:32.792536 17332 raft_consensus.cc:504] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.793866 17332 leader_election.cc:288] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647), f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
I20260924 10:37:32.798031 17607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.794307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:36121 (local address 127.2.193.130:40005)
0924 10:37:32.794428 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.794433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.794694 (+ 261us) server_negotiation.cc:407] Connection header received
0924 10:37:32.794839 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.794842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.794917 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.794994 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:32.795269 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.795747 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.796968 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.797431 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.797572 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.797621 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.797846 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.797921 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:37:32.798026 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.794221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:41155 (local address 127.2.193.131:36647)
0924 10:37:32.794378 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.794384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.794556 (+ 172us) server_negotiation.cc:407] Connection header received
0924 10:37:32.794748 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.794753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.794815 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.794891 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:32.795207 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.795741 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.796919 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.797415 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.797548 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.797605 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.797813 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.797892 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:37:32.798034 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.794193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:40005 (local address 127.2.193.129:36121)
0924 10:37:32.794565 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:32.794581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:32.794708 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:32.795034 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:32.795037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:32.795051 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:32.795126 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.795131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.795866 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.795872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.796799 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.796810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.797571 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.797575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.797671 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.797707 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:37:32.797811 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:37:32.797888 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":75,"thread_start_us":67,"threads_started":1}
I20260924 10:37:32.798044 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.794095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:36647 (local address 127.2.193.129:41155)
0924 10:37:32.794440 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:32.794456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:32.794589 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:32.794929 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:32.794932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:32.794955 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:32.795087 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.795093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.795868 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.795872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.796785 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:32.796795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.797544 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.797548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:32.797637 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.797699 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 10:37:32.797764 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 10:37:32.797869 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":70,"thread_start_us":91,"threads_started":1}
I20260924 10:37:32.798921 17418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f34383f65488b9a1fb0ca0346399b" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89b4cc25f9a4514933adfa2204c1a21" is_pre_election: true
I20260924 10:37:32.798919 17550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f34383f65488b9a1fb0ca0346399b" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ee8b456f8f4db3a13554a7146f9c65" is_pre_election: true
I20260924 10:37:32.799250 17418 raft_consensus.cc:2454] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 0.
I20260924 10:37:32.799260 17550 raft_consensus.cc:2454] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 0.
I20260924 10:37:32.799746 17213 leader_election.cc:302] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48ee8b456f8f4db3a13554a7146f9c65, 98ccb7f73e2c42dc910ea43f285527e1, f89b4cc25f9a4514933adfa2204c1a21; no voters:
I20260924 10:37:32.800026 17332 raft_consensus.cc:2781] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:37:32.800100 17332 raft_consensus.cc:482] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:37:32.800130 17332 raft_consensus.cc:3037] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:32.801297 17332 raft_consensus.cc:504] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.801673 17332 leader_election.cc:288] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 election: Requested vote from peers 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647), f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
I20260924 10:37:32.802052 17550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f34383f65488b9a1fb0ca0346399b" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ee8b456f8f4db3a13554a7146f9c65"
I20260924 10:37:32.802079 17418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9f34383f65488b9a1fb0ca0346399b" candidate_uuid: "98ccb7f73e2c42dc910ea43f285527e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89b4cc25f9a4514933adfa2204c1a21"
I20260924 10:37:32.802198 17550 raft_consensus.cc:3037] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:32.802212 17418 raft_consensus.cc:3037] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:32.803231 17550 raft_consensus.cc:2454] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 1.
I20260924 10:37:32.803289 17418 raft_consensus.cc:2454] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ccb7f73e2c42dc910ea43f285527e1 in term 1.
I20260924 10:37:32.803560 17213 leader_election.cc:302] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ee8b456f8f4db3a13554a7146f9c65, 98ccb7f73e2c42dc910ea43f285527e1; no voters:
I20260924 10:37:32.803727 17332 raft_consensus.cc:2781] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:32.803933 17332 raft_consensus.cc:686] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 LEADER]: Becoming Leader. State: Replica: 98ccb7f73e2c42dc910ea43f285527e1, State: Running, Role: LEADER
I20260924 10:37:32.804211 17332 consensus_queue.cc:237] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:32.807428 17134 catalog_manager.cc:5495] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98ccb7f73e2c42dc910ea43f285527e1 (127.2.193.129). New cstate: current_term: 1 leader_uuid: "98ccb7f73e2c42dc910ea43f285527e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } health_report { overall_health: HEALTHY } } }
I20260924 10:37:32.869057 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.865228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.2.193.129:42261)
0924 10:37:32.865561 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:32.865568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:32.865579 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:37:32.865656 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:32.865658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:32.865710 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:32.865842 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:37:32.866097 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.866646 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.867519 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:32.867755 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:32.867865 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:32.868174 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:37:32.868628 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:37:32.868753 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:32.868821 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 10:37:32.868881 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260924 10:37:32.873219 17550 raft_consensus.cc:1261] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Refusing update from remote peer 98ccb7f73e2c42dc910ea43f285527e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:32.873245 17418 raft_consensus.cc:1261] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Refusing update from remote peer 98ccb7f73e2c42dc910ea43f285527e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:32.873998 17332 consensus_queue.cc:1035] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:32.874209 17617 consensus_queue.cc:1035] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:32.881937 17620 mvcc.cc:205] Tried to move back new op lower bound from 7332848651761664000 to 7332848651488628736. Current Snapshot: MvccSnapshot[applied={T|T < 7332848651761664000}]
I20260924 10:37:32.882948 17621 mvcc.cc:205] Tried to move back new op lower bound from 7332848651761664000 to 7332848651488628736. Current Snapshot: MvccSnapshot[applied={T|T < 7332848651761664000}]
I20260924 10:37:32.893280 17132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:32.893478 17132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:32.895977 17132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Sending DeleteTablet for 3 replicas of tablet bd9f34383f65488b9a1fb0ca0346399b
I20260924 10:37:32.896761 17260 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f34383f65488b9a1fb0ca0346399b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:32 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52802
I20260924 10:37:32.896787 17398 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f34383f65488b9a1fb0ca0346399b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:32 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:55996
I20260924 10:37:32.896991 17530 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9f34383f65488b9a1fb0ca0346399b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:32 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:47836
I20260924 10:37:32.897321 17632 tablet_replica.cc:330] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: stopping tablet replica
I20260924 10:37:32.897437 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 17176
I20260924 10:37:32.897634 17633 tablet_replica.cc:330] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: stopping tablet replica
I20260924 10:37:32.898128 17633 raft_consensus.cc:2229] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:32.898454 17633 raft_consensus.cc:2258] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:32.899382 17632 raft_consensus.cc:2229] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:32.899724 17632 raft_consensus.cc:2258] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:32.900200 17633 ts_tablet_manager.cc:1905] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:32.901067 17632 ts_tablet_manager.cc:1905] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:32.903266 17632 ts_tablet_manager.cc:1918] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:32.903404 17632 log.cc:1196] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/bd9f34383f65488b9a1fb0ca0346399b
I20260924 10:37:32.903698 17632 ts_tablet_manager.cc:1939] T bd9f34383f65488b9a1fb0ca0346399b P f89b4cc25f9a4514933adfa2204c1a21: Deleting consensus metadata
I20260924 10:37:32.903925 17633 ts_tablet_manager.cc:1918] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:32.904042 17633 log.cc:1196] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/bd9f34383f65488b9a1fb0ca0346399b
I20260924 10:37:32.904367 17633 ts_tablet_manager.cc:1939] T bd9f34383f65488b9a1fb0ca0346399b P 48ee8b456f8f4db3a13554a7146f9c65: Deleting consensus metadata
W20260924 10:37:32.904840 17123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:42261 (error 108)
I20260924 10:37:32.905078 17124 catalog_manager.cc:4841] TS f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005): tablet bd9f34383f65488b9a1fb0ca0346399b (table test-table [id=2a4d6b480d9a42acbec85714c8907967]) successfully deleted
I20260924 10:37:32.905648 17123 catalog_manager.cc:4841] TS 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647): tablet bd9f34383f65488b9a1fb0ca0346399b (table test-table [id=2a4d6b480d9a42acbec85714c8907967]) successfully deleted
I20260924 10:37:32.906054 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 17331
I20260924 10:37:32.906317 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:32.906017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:42261 (local address 127.0.0.1:52812)
0924 10:37:32.906150 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:32.906221 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:42261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 10:37:32.906651 17123 catalog_manager.cc:4584] TS 98ccb7f73e2c42dc910ea43f285527e1 (127.2.193.129:42261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd9f34383f65488b9a1fb0ca0346399b: Network error: Client connection negotiation failed: client connection to 127.2.193.129:42261: connect: Connection refused (error 111)
I20260924 10:37:32.912158 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 17467
I20260924 10:37:32.917847 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 17105
I20260924 10:37:32.924237 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40973
--webserver_interface=127.2.193.190
--webserver_port=46649
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:33.037652 17635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:33.037942 17635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:33.037985 17635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:33.042276 17635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:33.042341 17635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:33.042361 17635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:33.042384 17635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:33.042400 17635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:33.047911 17635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=46649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:33.049016 17635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:33.050164 17635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:33.058787 17641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:33.058795 17643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:33.058789 17640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:33.059000 17635 server_base.cc:1034] running on GCE node
I20260924 10:37:33.059757 17635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:33.060957 17635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:33.062130 17635 hybrid_clock.cc:648] HybridClock initialized: now 1790246253062125 us; error 62 us; skew 500 ppm
Sep 24 10:37:33 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246253, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:33.065073 17635 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:33.066097 17635 webserver.cc:460] Webserver started at http://127.2.193.190:46649/ using document root <none> and password file <none>
I20260924 10:37:33.066674 17635 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:33.066740 17635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:33.070698 17635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:37:33.072924 17650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.074249 17635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:37:33.074415 17635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "abb8de7263974a189f65a19f9121fe63"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "2253e5e248a154434e89f68844aba554"
server_key_iv: "0562b6afd5e3cd4281185933da0a6299"
server_key_version: "encryptionkey@0"
I20260924 10:37:33.074998 17635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:33.101943 17635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:33.105046 17635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:33.105309 17635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:33.112811 17635 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:40973
I20260924 10:37:33.113114 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:40973 every 8 connection(s)
I20260924 10:37:33.114019 17635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:33.118413 17703 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:37:33.120107 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Bootstrap starting.
I20260924 10:37:33.121254 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 17635
I20260924 10:37:33.121757 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:42261
--local_ip_for_outbound_sockets=127.2.193.129
--tserver_master_addrs=127.2.193.190:40973
--webserver_port=38911
--webserver_interface=127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:37:33.138511 17703 log.cc:824] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:33.150233 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:37:33.150677 17703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63: Bootstrap complete.
I20260924 10:37:33.154865 17703 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:33.155260 17703 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb8de7263974a189f65a19f9121fe63, State: Initialized, Role: FOLLOWER
I20260924 10:37:33.155761 17703 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:33.155879 17703 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:37:33.155967 17703 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:37:33.156081 17703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:37:33.158244 17703 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:33.158619 17703 leader_election.cc:302] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb8de7263974a189f65a19f9121fe63; no voters:
I20260924 10:37:33.158990 17703 leader_election.cc:288] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:37:33.159078 17707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:37:33.159258 17707 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [term 3 LEADER]: Becoming Leader. State: Replica: abb8de7263974a189f65a19f9121fe63, State: Running, Role: LEADER
I20260924 10:37:33.159560 17707 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } }
I20260924 10:37:33.160285 17703 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:33.160974 17708 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "abb8de7263974a189f65a19f9121fe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } } }
I20260924 10:37:33.161108 17708 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:33.161978 17714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:33.162199 17709 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb8de7263974a189f65a19f9121fe63. Latest consensus state: current_term: 3 leader_uuid: "abb8de7263974a189f65a19f9121fe63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb8de7263974a189f65a19f9121fe63" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 40973 } } }
I20260924 10:37:33.162392 17709 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb8de7263974a189f65a19f9121fe63 [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:33.167480 17714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:33.168313 17714 catalog_manager.cc:1259] Loaded cluster ID: 7a359b5fd67a4b368956640c2024ce64
I20260924 10:37:33.168380 17714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 10:37:33.169459 17714 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f82832319ad google::LogMessage::Fail() at ??:0
@ 0x7f828323488a google::LogMessage::SendToLog() at ??:0
@ 0x7f82832314de google::LogMessage::Flush() at ??:0
@ 0x7f8283232a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f828db70fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f828db71d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f828db6e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f828dbb2640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f82854833d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8285483c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f82854853f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8285474512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8286ab86db start_thread at ??:0
@ 0x7f8281d2f61f clone at ??:0
W20260924 10:37:33.272820 17705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:33.273115 17705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:33.273161 17705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:33.277415 17705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:33.277483 17705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:33.277576 17705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:37:33.283186 17705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:42261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=38911
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:33.284384 17705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:33.285584 17705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:33.295758 17744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:33.295817 17747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:33.295967 17705 server_base.cc:1034] running on GCE node
W20260924 10:37:33.296015 17745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:33.296446 17705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:33.297145 17705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:33.298346 17705 hybrid_clock.cc:648] HybridClock initialized: now 1790246253298325 us; error 34 us; skew 500 ppm
Sep 24 10:37:33 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246253, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:33.301420 17705 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:37:33.302393 17705 webserver.cc:460] Webserver started at http://127.2.193.129:38911/ using document root <none> and password file <none>
I20260924 10:37:33.302999 17705 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:33.303097 17705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:33.307263 17705 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:37:33.309782 17754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.311208 17705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:37:33.311352 17705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "98ccb7f73e2c42dc910ea43f285527e1"
format_stamp: "Formatted at 2026-09-24 10:37:30 on dist-test-slave-8wc7"
server_key: "684321b61981c34e59bfda4ff1788a7e"
server_key_iv: "d080cfab4feafadf013f1208d66af08b"
server_key_version: "encryptionkey@0"
I20260924 10:37:33.311942 17705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:33.330705 17705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:33.334213 17705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:33.334441 17705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:33.335080 17705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:33.336238 17761 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:37:33.342027 17705 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:37:33.342119 17705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 10:37:33.342208 17705 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:37:33.343963 17705 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:37:33.344043 17705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:37:33.344357 17761 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap starting.
I20260924 10:37:33.355005 17705 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:42261
I20260924 10:37:33.355273 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:42261 every 8 connection(s)
I20260924 10:37:33.356074 17705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:37:33.359959 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 17705
I20260924 10:37:33.360379 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:40005
--local_ip_for_outbound_sockets=127.2.193.130
--tserver_master_addrs=127.2.193.190:40973
--webserver_port=43561
--webserver_interface=127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:37:33.367182 17761 log.cc:824] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:33.376992 17761 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:37:33.377492 17761 tablet_bootstrap.cc:492] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Bootstrap complete.
I20260924 10:37:33.378940 17761 ts_tablet_manager.cc:1397] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.010s
I20260924 10:37:33.382467 17761 raft_consensus.cc:348] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:33.383020 17761 raft_consensus.cc:729] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ccb7f73e2c42dc910ea43f285527e1, State: Initialized, Role: FOLLOWER
I20260924 10:37:33.383565 17761 consensus_queue.cc:260] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:33.384662 17761 ts_tablet_manager.cc:1428] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260924 10:37:33.402529 17871 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:37:33.358143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:58359)
0924 10:37:33.358630 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.358685 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:33.359507 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:33.401740 (+ 42233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
W20260924 10:37:33.403069 17869 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:40973 (0 consecutive failures): Network error: Failed to ping master at 127.2.193.190:40973: Client connection negotiation failed: client connection to 127.2.193.190:40973: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:37:33.403853 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.403597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:47965)
0924 10:37:33.403723 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.403787 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:37:33.404569 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.404364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:50763)
0924 10:37:33.404469 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.404522 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:37:33.404752 17869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:37:33.478564 17872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:33.478906 17872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:33.478992 17872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:33.483134 17872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:33.483242 17872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:33.483342 17872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:37:33.488777 17872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:40005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=43561
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:33.489955 17872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:33.491048 17872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:33.501003 17880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:33.501052 17879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:33.501030 17872 server_base.cc:1034] running on GCE node
W20260924 10:37:33.501049 17882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:33.501694 17872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:33.502377 17872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:33.503575 17872 hybrid_clock.cc:648] HybridClock initialized: now 1790246253503550 us; error 62 us; skew 500 ppm
Sep 24 10:37:33 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246253, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:33.506475 17872 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:37:33.507483 17872 webserver.cc:460] Webserver started at http://127.2.193.130:43561/ using document root <none> and password file <none>
I20260924 10:37:33.508095 17872 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:33.508193 17872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:33.512342 17872 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:37:33.514598 17889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.516029 17872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:37:33.516175 17872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "f89b4cc25f9a4514933adfa2204c1a21"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "bd5316cd9af72bbfaee9a8afc985834d"
server_key_iv: "9d494a9b31093d15085f395ee7b99985"
server_key_version: "encryptionkey@0"
I20260924 10:37:33.516757 17872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:33.527371 17872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:33.530344 17872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:33.530556 17872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:33.531159 17872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:33.531998 17872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:33.532069 17872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.532145 17872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:33.532186 17872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.541424 17872 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:40005
I20260924 10:37:33.541443 18002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:40005 every 8 connection(s)
I20260924 10:37:33.542501 17872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:37:33.546418 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.544708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:36667)
0924 10:37:33.545224 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.545668 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
W20260924 10:37:33.546859 18003 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:40973 (0 consecutive failures): Network error: Failed to ping master at 127.2.193.190:40973: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
I20260924 10:37:33.547523 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.547259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:40695)
0924 10:37:33.547390 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.547446 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:37:33.547793 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 17872
I20260924 10:37:33.548171 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.547959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:47517)
0924 10:37:33.548070 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.548118 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 10:37:33.548229 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:36647
--local_ip_for_outbound_sockets=127.2.193.131
--tserver_master_addrs=127.2.193.190:40973
--webserver_port=37315
--webserver_interface=127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:33.548533 18003 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:37:33.662359 18006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:33.662673 18006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:33.662719 18006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:33.667164 18006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:33.667244 18006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:33.667306 18006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:37:33.672847 18006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:36647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=37315
--tserver_master_addrs=127.2.193.190:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:33.674041 18006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:33.675144 18006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:33.684958 18011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:33.684958 18012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:33.684962 18014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:33.685395 18006 server_base.cc:1034] running on GCE node
I20260924 10:37:33.685891 18006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:33.686586 18006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:33.687780 18006 hybrid_clock.cc:648] HybridClock initialized: now 1790246253687763 us; error 30 us; skew 500 ppm
Sep 24 10:37:33 dist-test-slave-8wc7 krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246253, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:33.690730 18006 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:37:33.691747 18006 webserver.cc:460] Webserver started at http://127.2.193.131:37315/ using document root <none> and password file <none>
I20260924 10:37:33.692364 18006 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:33.692457 18006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:33.696408 18006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:37:33.698731 18021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.699965 18006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:37:33.700102 18006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "48ee8b456f8f4db3a13554a7146f9c65"
format_stamp: "Formatted at 2026-09-24 10:37:31 on dist-test-slave-8wc7"
server_key: "beaa1b2361c85944d4d8ae110c0b775c"
server_key_iv: "ab0d0f8ee4ab70e8c75ba9c6c5384da1"
server_key_version: "encryptionkey@0"
I20260924 10:37:33.700685 18006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:33.726435 18006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:33.730090 18006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:33.730309 18006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:33.730955 18006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:33.731856 18006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:33.731930 18006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.732002 18006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:33.732048 18006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:33.741504 18006 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:36647
I20260924 10:37:33.741524 18134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:36647 every 8 connection(s)
I20260924 10:37:33.742935 18006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:37:33.744918 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18006
I20260924 10:37:33.747264 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.745545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:46281)
0924 10:37:33.746059 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.746533 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
W20260924 10:37:33.747759 18135 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:40973 (0 consecutive failures): Network error: Failed to ping master at 127.2.193.190:40973: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
I20260924 10:37:33.748301 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.748088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:58013)
0924 10:37:33.748201 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.748247 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:37:33.748939 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:33.748749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:43477)
0924 10:37:33.748848 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:33.748888 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:37:33.749125 18135 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:37:34.405953 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:34.405493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:46267)
0924 10:37:34.405772 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:34.405837 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260924 10:37:34.549732 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:34.549229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:51895)
0924 10:37:34.549516 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:34.549584 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":69,"threads_started":1}
I20260924 10:37:34.584304 18141 raft_consensus.cc:482] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:34.584540 18141 raft_consensus.cc:504] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:34.585824 18141 leader_election.cc:288] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647), f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
Sep 24 10:37:34 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246253, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.131@KRBTEST.COM
Sep 24 10:37:34 dist-test-slave-8wc7 krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246253, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.130@KRBTEST.COM
I20260924 10:37:34.600332 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:34.585994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:36647 (local address 127.2.193.129:46291)
0924 10:37:34.586166 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:34.586183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:34.586320 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:34.588504 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:34.588510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:34.588879 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:34.589444 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:34.589460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.590776 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.590781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:34.591840 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:34.591850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.592682 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.592686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:34.592915 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:34.593659 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:34.593678 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:34.595433 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:34.597982 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:34.597989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:34.598001 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:34.598248 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:34.598646 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:34.598648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:34.598649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:34.598770 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:34.599589 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:34.599594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:34.599806 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 10:37:34.599961 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:37:34.600194 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":109,"spinlock_wait_cycles":1920}
I20260924 10:37:34.600489 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:34.586220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:46291 (local address 127.2.193.131:36647)
0924 10:37:34.586592 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:34.586610 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:34.586656 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:37:34.587600 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:34.587610 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:34.588089 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:34.588434 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:34.589626 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.590613 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.591971 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.592545 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.592731 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:34.595660 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:34.595665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:34.595678 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:34.595705 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:34.597866 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:34.598382 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:34.598389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:34.598394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:34.598519 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:34.598869 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:34.598875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:34.598878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:34.599480 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:34.599622 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:34.600081 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 10:37:34.600312 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260924 10:37:34.601033 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:34.586086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:40005 (local address 127.2.193.129:46043)
0924 10:37:34.586443 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:34.586459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:34.586569 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:34.588937 (+ 2368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:34.588940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:34.589104 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:34.589443 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:34.589460 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.590781 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.590784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:34.591841 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:34.591851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.592662 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.592666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:34.592912 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:34.593611 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:34.593643 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:34.595753 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:34.598343 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:34.598345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:34.598346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:34.598505 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:34.598883 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:34.598885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:34.598886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:34.598941 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:34.600552 (+ 1611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:34.600557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:34.600655 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 10:37:34.600749 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:37:34.600868 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":159,"spinlock_wait_cycles":10752,"thread_start_us":94,"threads_started":1}
I20260924 10:37:34.601572 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:34.586424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:46043 (local address 127.2.193.130:40005)
0924 10:37:34.586764 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:34.586785 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:34.586815 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:37:34.587600 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:34.587610 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:34.588089 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:34.588434 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:34.589645 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.590626 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.591966 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:34.592538 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:34.592717 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:34.595952 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:34.595960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:34.595977 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:34.596019 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:34.597870 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:34.598636 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:34.598642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:34.598650 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:34.598776 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:34.599022 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:34.599025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:34.599026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:34.599485 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:34.599622 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:34.601087 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0924 10:37:34.601405 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":128,"threads_started":1}
W20260924 10:37:34.603196 17756 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
W20260924 10:37:34.603627 17757 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
I20260924 10:37:34.603701 17757 leader_election.cc:302] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98ccb7f73e2c42dc910ea43f285527e1; no voters: 48ee8b456f8f4db3a13554a7146f9c65, f89b4cc25f9a4514933adfa2204c1a21
I20260924 10:37:34.603947 18141 raft_consensus.cc:2726] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:37:34.750099 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:34.749699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:41311)
0924 10:37:34.749948 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:34.750002 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260924 10:37:35.407327 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:35.406798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:57739)
0924 10:37:35.407127 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:35.407206 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
I20260924 10:37:35.550990 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:35.550517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:53381)
0924 10:37:35.550811 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:35.550884 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260924 10:37:35.751538 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:35.750954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:38731)
0924 10:37:35.751291 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:35.751383 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
I20260924 10:37:36.408506 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:36.408067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:54829)
0924 10:37:36.408316 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:36.408376 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260924 10:37:36.552223 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:36.551808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:41277)
0924 10:37:36.552058 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:36.552112 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260924 10:37:36.753041 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:36.752478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:34505)
0924 10:37:36.752840 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:36.752917 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":96,"threads_started":1}
I20260924 10:37:36.791260 18152 raft_consensus.cc:482] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:36.791388 18152 raft_consensus.cc:504] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:36.791904 18152 leader_election.cc:288] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647), f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
W20260924 10:37:36.792802 17756 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
W20260924 10:37:36.792963 17757 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
I20260924 10:37:36.793082 17757 leader_election.cc:302] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98ccb7f73e2c42dc910ea43f285527e1; no voters: 48ee8b456f8f4db3a13554a7146f9c65, f89b4cc25f9a4514933adfa2204c1a21
I20260924 10:37:36.793303 18152 raft_consensus.cc:2726] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:37:37.409866 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:37.409420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:32787)
0924 10:37:37.409713 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:37.409770 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260924 10:37:37.553573 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:37.553124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:52565)
0924 10:37:37.553410 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:37.553466 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
I20260924 10:37:37.754361 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:37.753937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:51729)
0924 10:37:37.754200 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:37.754262 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260924 10:37:38.411327 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:38.410862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:49655)
0924 10:37:38.411167 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:38.411229 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
I20260924 10:37:38.554943 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:38.554475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:46267)
0924 10:37:38.554766 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:38.554831 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260924 10:37:38.755556 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:38.755144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:33739)
0924 10:37:38.755397 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:38.755452 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260924 10:37:39.412670 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:39.412192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:46867)
0924 10:37:39.412499 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:39.412567 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260924 10:37:39.556325 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:39.555832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:42913)
0924 10:37:39.556137 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:39.556206 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20260924 10:37:39.756989 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:39.756488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:34913)
0924 10:37:39.756802 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:39.756868 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
I20260924 10:37:40.049810 18162 raft_consensus.cc:482] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:40.049968 18162 raft_consensus.cc:504] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:40.050421 18162 leader_election.cc:288] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647), f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
W20260924 10:37:40.051206 17757 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
W20260924 10:37:40.051301 17756 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
I20260924 10:37:40.051419 17756 leader_election.cc:302] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98ccb7f73e2c42dc910ea43f285527e1; no voters: 48ee8b456f8f4db3a13554a7146f9c65, f89b4cc25f9a4514933adfa2204c1a21
I20260924 10:37:40.051584 18162 raft_consensus.cc:2726] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:37:40.413964 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:40.413529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:33793)
0924 10:37:40.413799 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:40.413861 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260924 10:37:40.557650 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:40.557191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:47471)
0924 10:37:40.557480 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:40.557553 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260924 10:37:40.758333 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:40.757846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:60437)
0924 10:37:40.758171 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:40.758236 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":78,"threads_started":1}
I20260924 10:37:41.415341 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:41.414886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:59115)
0924 10:37:41.415175 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:41.415239 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260924 10:37:41.558915 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:41.558492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:57527)
0924 10:37:41.558756 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:41.558817 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260924 10:37:41.759740 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:41.759240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:43215)
0924 10:37:41.759568 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:41.759638 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260924 10:37:42.416464 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:42.416061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:36443)
0924 10:37:42.416305 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:42.416362 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260924 10:37:42.560160 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:42.559732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:44415)
0924 10:37:42.559987 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:42.560051 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260924 10:37:42.761089 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:42.760606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:56863)
0924 10:37:42.760924 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:42.760990 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
I20260924 10:37:43.417822 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:43.417388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:46167)
0924 10:37:43.417664 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:43.417723 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260924 10:37:43.561614 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:43.561044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:56141)
0924 10:37:43.561437 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:43.561511 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":78,"threads_started":1}
I20260924 10:37:43.762431 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:43.761986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:34337)
0924 10:37:43.762266 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:43.762330 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260924 10:37:44.419057 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:44.418572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:53057)
0924 10:37:44.418882 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:44.418950 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":91,"threads_started":1}
I20260924 10:37:44.562832 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:44.562405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:53321)
0924 10:37:44.562681 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:44.562737 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20260924 10:37:44.763736 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:44.763279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:49787)
0924 10:37:44.763561 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:44.763627 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260924 10:37:45.102039 18178 raft_consensus.cc:482] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:45.102167 18178 raft_consensus.cc:504] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48ee8b456f8f4db3a13554a7146f9c65" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36647 } } peers { permanent_uuid: "f89b4cc25f9a4514933adfa2204c1a21" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 40005 } } peers { permanent_uuid: "98ccb7f73e2c42dc910ea43f285527e1" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42261 } }
I20260924 10:37:45.102644 18178 leader_election.cc:288] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647), f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005)
W20260924 10:37:45.103478 17756 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48ee8b456f8f4db3a13554a7146f9c65 (127.2.193.131:36647): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
W20260924 10:37:45.103652 17757 leader_election.cc:341] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f89b4cc25f9a4514933adfa2204c1a21 (127.2.193.130:40005): Not found: Tablet not found: bd9f34383f65488b9a1fb0ca0346399b
I20260924 10:37:45.103740 17757 leader_election.cc:302] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98ccb7f73e2c42dc910ea43f285527e1; no voters: 48ee8b456f8f4db3a13554a7146f9c65, f89b4cc25f9a4514933adfa2204c1a21
I20260924 10:37:45.103888 18178 raft_consensus.cc:2726] T bd9f34383f65488b9a1fb0ca0346399b P 98ccb7f73e2c42dc910ea43f285527e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:37:45.420315 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:45.419833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:46961)
0924 10:37:45.420148 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:45.420215 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":88,"threads_started":1}
I20260924 10:37:45.564146 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:45.563693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:33661)
0924 10:37:45.563986 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:45.564049 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260924 10:37:45.764966 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:45.764504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:37501)
0924 10:37:45.764796 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:45.764854 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260924 10:37:46.421686 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:46.421164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:37547)
0924 10:37:46.421489 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:46.421575 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260924 10:37:46.565496 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:46.564945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:53281)
0924 10:37:46.565320 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:46.565391 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":95,"threads_started":1}
I20260924 10:37:46.766335 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:46.765866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:43637)
0924 10:37:46.766168 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:46.766233 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260924 10:37:47.422910 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:47.422464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:40171)
0924 10:37:47.422755 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:47.422814 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":77,"threads_started":1}
I20260924 10:37:47.566840 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:47.566355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:41739)
0924 10:37:47.566681 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:47.566744 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":122,"threads_started":1}
I20260924 10:37:47.767664 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:47.767217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.131:41781)
0924 10:37:47.767510 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:47.767564 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":125,"threads_started":1}
I20260924 10:37:48.424474 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:48.423971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.129:36931)
0924 10:37:48.424269 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:48.424323 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":125,"threads_started":1}
I20260924 10:37:48.568188 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:48.567805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:40973 (local address 127.2.193.130:40185)
0924 10:37:48.568036 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:48.568088 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:40973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260924 10:37:48.754006 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 17705
I20260924 10:37:48.759796 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 17872
I20260924 10:37:48.765321 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18006
2026-09-24T10:37:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19565 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:37:48 dist-test-slave-8wc7 krb5kdc[18192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:37:48 dist-test-slave-8wc7 krb5kdc[18192](info): set up 2 sockets
Sep 24 10:37:48 dist-test-slave-8wc7 krb5kdc[18192](info): commencing operation
krb5kdc: starting...
W20260924 10:37:49.809335 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:37:49 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:37:49.934206 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34615
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:50.047742 18205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:50.048049 18205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:50.048092 18205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:50.052475 18205 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:37:50.052546 18205 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:37:50.052562 18205 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:37:50.052575 18205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:37:50.052594 18205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:50.052659 18205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:50.052687 18205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:37:50.052721 18205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:37:50.058277 18205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:50.059404 18205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:50.060542 18205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:50.069815 18210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:50.069852 18211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:50.069850 18205 server_base.cc:1034] running on GCE node
W20260924 10:37:50.069890 18213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:50.070715 18205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:50.071928 18205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:50.073112 18205 hybrid_clock.cc:648] HybridClock initialized: now 1790246270073093 us; error 37 us; skew 500 ppm
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:50.076152 18205 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:37:50.077160 18205 webserver.cc:460] Webserver started at http://127.2.193.190:34945/ using document root <none> and password file <none>
I20260924 10:37:50.077785 18205 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:50.077857 18205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:50.078032 18205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:50.080098 18205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "a9ec9b62965a4c71ab27fe700619256c"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "077b80fa86743ed91cfe13932062ea1d"
server_key_iv: "7f60a2e47f0697c6246da1be3c564cd1"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.080673 18205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "a9ec9b62965a4c71ab27fe700619256c"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "077b80fa86743ed91cfe13932062ea1d"
server_key_iv: "7f60a2e47f0697c6246da1be3c564cd1"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.084506 18205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 10:37:50.087119 18220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.088445 18205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:37:50.088610 18205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "a9ec9b62965a4c71ab27fe700619256c"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "077b80fa86743ed91cfe13932062ea1d"
server_key_iv: "7f60a2e47f0697c6246da1be3c564cd1"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.088734 18205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:50.111595 18205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:50.115254 18205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:50.115485 18205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:50.123018 18205 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:34615
I20260924 10:37:50.123317 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:34615 every 8 connection(s)
I20260924 10:37:50.124081 18205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:37:50.126963 18273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:50.131424 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18205
I20260924 10:37:50.131580 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:37:50.131867 2822 external_mini_cluster.cc:1428] Setting key 2d51aad0ac5e14f336d439b90a48c037
I20260924 10:37:50.132992 18273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Bootstrap starting.
I20260924 10:37:50.135450 18273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:50.136400 18273 log.cc:824] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:50.138707 18273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: No bootstrap required, opened a new log
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:50.143204 18273 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:37:50.143477 18273 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:50.143569 18273 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ec9b62965a4c71ab27fe700619256c, State: Initialized, Role: FOLLOWER
I20260924 10:37:50.144048 18273 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:37:50.144207 18273 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:37:50.144295 18273 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:37:50.144426 18273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:50.145598 18273 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:37:50.145958 18273 leader_election.cc:302] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9ec9b62965a4c71ab27fe700619256c; no voters:
I20260924 10:37:50.146309 18273 leader_election.cc:288] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:37:50.146600 18278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:50.146852 18278 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 LEADER]: Becoming Leader. State: Replica: a9ec9b62965a4c71ab27fe700619256c, State: Running, Role: LEADER
I20260924 10:37:50.147243 18278 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:37:50.147687 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.133378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.2.193.190:34615)
0924 10:37:50.133810 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.133826 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.133869 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:37:50.134525 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.134542 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.134899 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.135203 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:50.136171 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.137321 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.138501 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.139069 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.139202 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.142117 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:50.142122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:50.142137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:50.142166 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:50.144186 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.144788 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.144793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.144798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.144916 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.145260 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.145262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.145263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.145722 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:50.145936 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.146265 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.146661 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260924 10:37:50.147854 18273 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:37:50.149022 18280 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9ec9b62965a4c71ab27fe700619256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } } }
I20260924 10:37:50.149267 18280 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:50.149618 18280 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ec9b62965a4c71ab27fe700619256c. Latest consensus state: current_term: 1 leader_uuid: "a9ec9b62965a4c71ab27fe700619256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } } }
I20260924 10:37:50.149765 18280 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: This master's current role is: LEADER
I20260924 10:37:50.150714 18288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:37:50.153532 18288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:37:50.159354 18288 catalog_manager.cc:1347] Generated new cluster ID: fe0d5dbf67624df8a016c9ed4e00f9d5
I20260924 10:37:50.159458 18288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:37:50.171849 18288 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:37:50.172899 18288 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:37:50.188555 18288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Generated new TSK 0
I20260924 10:37:50.189388 18288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:37:50.225502 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34615
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:50.345398 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:50.345695 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:50.345741 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:50.350317 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:50.350399 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:50.350493 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:37:50.356037 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:50.357239 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:50.358340 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:50.368495 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:50.368479 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:50.368640 18299 server_base.cc:1034] running on GCE node
W20260924 10:37:50.368788 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:50.369285 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:50.369900 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:50.371070 18299 hybrid_clock.cc:648] HybridClock initialized: now 1790246270371051 us; error 29 us; skew 500 ppm
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:50.374145 18299 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:37:50.375007 18299 webserver.cc:460] Webserver started at http://127.2.193.129:44221/ using document root <none> and password file <none>
I20260924 10:37:50.375589 18299 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:50.375651 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:50.375897 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:50.378113 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "a1b8e3edd5164198921549580e7590d8"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "f114873d6aa04e4c5aa008ba471541e7"
server_key_iv: "63a1b6dd40bd24a14e112c043cb92125"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.378695 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "a1b8e3edd5164198921549580e7590d8"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "f114873d6aa04e4c5aa008ba471541e7"
server_key_iv: "63a1b6dd40bd24a14e112c043cb92125"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.382434 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:37:50.384855 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.386147 18299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:37:50.386269 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "a1b8e3edd5164198921549580e7590d8"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "f114873d6aa04e4c5aa008ba471541e7"
server_key_iv: "63a1b6dd40bd24a14e112c043cb92125"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.386379 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:50.397816 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:50.400727 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:50.400938 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:50.401655 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:50.402541 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:50.402591 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.402631 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:50.402673 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.412367 18299 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38837
I20260924 10:37:50.412433 18427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38837 every 8 connection(s)
I20260924 10:37:50.413455 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:37:50.422731 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18299
I20260924 10:37:50.422889 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:37:50.423210 2822 external_mini_cluster.cc:1428] Setting key db3ead17408a6466708a22906d3f6bcd
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:50.428243 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.415380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:41163 (local address 127.2.193.190:34615)
0924 10:37:50.415532 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.415536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.416379 (+ 843us) server_negotiation.cc:407] Connection header received
0924 10:37:50.417833 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.417837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.417894 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.418017 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:50.419211 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.419774 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.420939 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.421262 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.421376 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.424125 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:50.424129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:50.424131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:50.424157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:50.425811 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.426493 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.426498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.426501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.426604 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.426943 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.426948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.426949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.427215 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:50.427323 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.427830 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.427990 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:37:50.429026 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.415769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:41163)
0924 10:37:50.416224 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:50.416269 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:50.417608 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:50.418197 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:50.418204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:50.418574 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:50.419060 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.419074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.419909 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.419917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.420807 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.420816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.421417 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.421421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.421601 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.422157 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:50.422173 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:50.423926 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:50.425933 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:50.425942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:50.425962 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:50.426368 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:50.426703 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:50.426705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:50.426706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:50.426849 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:50.427317 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:50.427322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:50.427585 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 10:37:50.427749 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 10:37:50.428025 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
I20260924 10:37:50.430393 18428 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34615
I20260924 10:37:50.430675 18428 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:50.431193 18428 heartbeater.cc:511] Master 127.2.193.190:34615 requested a full tablet report, sending...
I20260924 10:37:50.433004 18237 ts_manager.cc:194] Registered new tserver with Master: a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837)
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:37:50.434432 18237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:41163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:37:50.452440 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34615
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:50.566808 18433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:50.567086 18433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:50.567160 18433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:50.571453 18433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:50.571518 18433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:50.571611 18433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:37:50.577039 18433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:50.578300 18433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:50.579370 18433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:37:50.589493 18433 server_base.cc:1034] running on GCE node
W20260924 10:37:50.589450 18441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:50.589511 18438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:50.589643 18439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:50.590132 18433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:50.590845 18433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:50.592003 18433 hybrid_clock.cc:648] HybridClock initialized: now 1790246270591987 us; error 27 us; skew 500 ppm
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:50.594956 18433 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:37:50.595783 18433 webserver.cc:460] Webserver started at http://127.2.193.130:43911/ using document root <none> and password file <none>
I20260924 10:37:50.596320 18433 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:50.596380 18433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:50.596545 18433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:50.598670 18433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "e6186b1c0f4342fd8b3a7b109a4c443a"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "21b3c880842e449ebb56f28db9600cc7"
server_key_iv: "7587c183783d13b68b534a9eba3998ed"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.599187 18433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "e6186b1c0f4342fd8b3a7b109a4c443a"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "21b3c880842e449ebb56f28db9600cc7"
server_key_iv: "7587c183783d13b68b534a9eba3998ed"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.602866 18433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:37:50.605297 18448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.606650 18433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:37:50.606760 18433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "e6186b1c0f4342fd8b3a7b109a4c443a"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "21b3c880842e449ebb56f28db9600cc7"
server_key_iv: "7587c183783d13b68b534a9eba3998ed"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.606864 18433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:50.622187 18433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:50.625041 18433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:50.625283 18433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:50.625842 18433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:50.626626 18433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:50.626672 18433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.626713 18433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:50.626734 18433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.635636 18433 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:38257
I20260924 10:37:50.635655 18561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:38257 every 8 connection(s)
I20260924 10:37:50.636590 18433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:37:50.639233 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18433
I20260924 10:37:50.639370 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:37:50.639642 2822 external_mini_cluster.cc:1428] Setting key 0b99e2aaae046eb4917cd8a7934a26ed
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:37:50.652511 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.638509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:38391 (local address 127.2.193.190:34615)
0924 10:37:50.638669 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.638674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.639693 (+ 1019us) server_negotiation.cc:407] Connection header received
0924 10:37:50.641208 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.641211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.641272 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.641340 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:50.643055 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.643951 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.645141 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.645613 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.645749 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.648591 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:50.648595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:50.648599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:50.648635 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:50.650385 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.651006 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.651009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.651010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.651087 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.651408 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.651411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.651412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.651600 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:50.651673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.652185 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.652338 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
I20260924 10:37:50.653587 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.639014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:38391)
0924 10:37:50.639509 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:50.639566 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:50.640979 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:50.641546 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:50.641557 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:50.642118 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:50.642864 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.642886 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.644110 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.644113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.645007 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.645015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.645772 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.645775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.645925 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.646568 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:50.646585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:50.648393 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:50.650549 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:50.650557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:50.650577 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:50.650902 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:50.651180 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:50.651182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:50.651184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:50.651318 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:50.651702 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:50.651707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:50.651974 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 10:37:50.652136 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 10:37:50.652478 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":145,"threads_started":1}
I20260924 10:37:50.654824 18562 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34615
I20260924 10:37:50.655107 18562 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:50.655579 18562 heartbeater.cc:511] Master 127.2.193.190:34615 requested a full tablet report, sending...
I20260924 10:37:50.656613 18237 ts_manager.cc:194] Registered new tserver with Master: e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
I20260924 10:37:50.657260 18237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:38391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:37:50.670195 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34615
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:37:50.785027 18567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:37:50.785513 18567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:37:50.785604 18567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:37:50.790088 18567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:37:50.790180 18567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:37:50.790269 18567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:37:50.796241 18567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:37:50.797488 18567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:37:50.798581 18567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:37:50.809489 18573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:50.809525 18572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:37:50.809818 18575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:37:50.809983 18567 server_base.cc:1034] running on GCE node
I20260924 10:37:50.810434 18567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:37:50.811245 18567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:37:50.812450 18567 hybrid_clock.cc:648] HybridClock initialized: now 1790246270812431 us; error 32 us; skew 500 ppm
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:37:50.815299 18567 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:37:50.816246 18567 webserver.cc:460] Webserver started at http://127.2.193.131:46249/ using document root <none> and password file <none>
I20260924 10:37:50.816788 18567 fs_manager.cc:359] Metadata directory not provided
I20260924 10:37:50.816844 18567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:37:50.817013 18567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:37:50.819150 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "a933650b6eb3423686d3838d36c07b13"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "c7ef2bd91cd70b901c3926b6dff13c05"
server_key_iv: "5df72e4c18596e599a5ba8764c8bfca9"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.819696 18567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "a933650b6eb3423686d3838d36c07b13"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "c7ef2bd91cd70b901c3926b6dff13c05"
server_key_iv: "5df72e4c18596e599a5ba8764c8bfca9"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.823498 18567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:37:50.826102 18582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.827441 18567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:37:50.827544 18567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "a933650b6eb3423686d3838d36c07b13"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "c7ef2bd91cd70b901c3926b6dff13c05"
server_key_iv: "5df72e4c18596e599a5ba8764c8bfca9"
server_key_version: "encryptionkey@0"
I20260924 10:37:50.827643 18567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:37:50.838011 18567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:37:50.841023 18567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:37:50.841254 18567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:37:50.841836 18567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:37:50.842629 18567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:37:50.842675 18567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.842716 18567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:37:50.842731 18567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:37:50.851745 18567 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:42249
I20260924 10:37:50.851763 18695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:42249 every 8 connection(s)
I20260924 10:37:50.852689 18567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:37:50.856694 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18567
I20260924 10:37:50.856817 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:37:50.857148 2822 external_mini_cluster.cc:1428] Setting key edc501f336fd21ba36130c9cf5db162f
Sep 24 10:37:50 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:37:50.866230 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.854553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:50667 (local address 127.2.193.190:34615)
0924 10:37:50.854707 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.854710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.855576 (+ 866us) server_negotiation.cc:407] Connection header received
0924 10:37:50.857041 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.857044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.857092 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.857213 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:50.858657 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.859261 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.860441 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.860664 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.860743 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.863101 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:50.863105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:50.863107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:50.863132 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:50.864384 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.864853 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.864855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.864857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.864927 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.865267 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.865270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.865271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.865442 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:50.865522 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.865999 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.866105 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:37:50.867101 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.854938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:50667)
0924 10:37:50.855408 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:50.855455 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:50.856783 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:50.857425 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:50.857435 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:50.857826 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:37:50.858523 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.858540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.859421 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.859426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.860325 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.860335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.860793 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.860795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.860945 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.861486 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:37:50.861502 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:37:50.862941 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:37:50.864493 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:50.864497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:50.864510 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:50.864768 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:50.865004 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:37:50.865006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:37:50.865007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:37:50.865136 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:37:50.865539 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:37:50.865544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:37:50.865783 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 10:37:50.865936 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 10:37:50.866196 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260924 10:37:50.868265 18696 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34615
I20260924 10:37:50.868535 18696 heartbeater.cc:464] Registering TS with master...
I20260924 10:37:50.869045 18696 heartbeater.cc:511] Master 127.2.193.190:34615 requested a full tablet report, sending...
I20260924 10:37:50.870273 18237 ts_manager.cc:194] Registered new tserver with Master: a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249)
I20260924 10:37:50.870867 18237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:50667
I20260924 10:37:50.871348 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:37:50.880981 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.873584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.2.193.190:34615)
0924 10:37:50.873727 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.873732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.873931 (+ 199us) server_negotiation.cc:407] Connection header received
0924 10:37:50.874027 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.874029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.874068 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.874134 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:37:50.874794 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.875296 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.876083 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.876288 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.876365 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.877748 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:37:50.877751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:37:50.877753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:37:50.877776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:37:50.879304 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.879780 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.879783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.879784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.879859 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:37:50.880093 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:37:50.880095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:37:50.880096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:37:50.880306 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:37:50.880408 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.880669 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.880766 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:37:50.884091 18237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:37:50.887305 18237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:37:50.902177 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.896383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.2.193.129:38837)
0924 10:37:50.896818 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.896835 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.896860 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:37:50.896942 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.896950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.897207 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.897382 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:50.897712 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.898865 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.900587 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.901457 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.901575 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.901737 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.901910 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.902010 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":140,"threads_started":1}
I20260924 10:37:50.902166 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.896247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38837 (local address 127.0.0.1:37204)
0924 10:37:50.896618 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:50.896676 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:50.896809 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:50.897390 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:50.897401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:50.897432 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:50.897583 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.897591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.899011 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.899016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.900448 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.900457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.901579 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.901583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.901703 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.901739 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:37:50.901872 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:37:50.902010 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":29,"thread_start_us":78,"threads_started":1}
I20260924 10:37:50.902750 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.897319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:42249 (local address 127.0.0.1:52378)
0924 10:37:50.898156 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:50.898169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:50.898262 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:50.898868 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:50.898871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:50.898887 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:50.898996 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.899004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.900307 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.900311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.901152 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.901161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.902260 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.902263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.902358 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.902391 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:37:50.902437 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 10:37:50.902520 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":94,"threads_started":1}
I20260924 10:37:50.903195 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.896974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:38257 (local address 127.0.0.1:41460)
0924 10:37:50.897571 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:50.897584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:50.897692 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:50.898421 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:50.898424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:50.898440 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:50.898533 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.898538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.900151 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.900154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.901374 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:50.901386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.902426 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.902430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:50.902517 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.902898 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 10:37:50.902968 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:37:50.903053 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":21,"thread_start_us":101,"threads_started":1}
I20260924 10:37:50.903363 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.898101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.2.193.131:42249)
0924 10:37:50.898500 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.898510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.898527 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:37:50.898588 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.898592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.898732 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.898892 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:50.899123 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.900174 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.901304 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.902112 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.902938 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.903087 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.903130 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.903210 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260924 10:37:50.903537 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:50.897629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.2.193.130:38257)
0924 10:37:50.897971 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:50.897990 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:50.898014 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:37:50.898086 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:50.898091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:50.898267 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:50.898668 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:50.898831 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.900015 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.901504 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:50.902310 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:50.902409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:50.902553 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:50.903011 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 10:37:50.903112 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":138,"threads_started":1}
I20260924 10:37:50.906068 18630 tablet_service.cc:1467] Processing CreateTablet for tablet 00cfd035402d4b96a432d2b2408299ad (DEFAULT_TABLE table=test-table [id=3248267ddbc7436b979317997f3707ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:50.906116 18362 tablet_service.cc:1467] Processing CreateTablet for tablet 00cfd035402d4b96a432d2b2408299ad (DEFAULT_TABLE table=test-table [id=3248267ddbc7436b979317997f3707ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:50.906317 18496 tablet_service.cc:1467] Processing CreateTablet for tablet 00cfd035402d4b96a432d2b2408299ad (DEFAULT_TABLE table=test-table [id=3248267ddbc7436b979317997f3707ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:37:50.907168 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cfd035402d4b96a432d2b2408299ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:50.907168 18496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cfd035402d4b96a432d2b2408299ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:50.907184 18630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00cfd035402d4b96a432d2b2408299ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:37:50.913753 18713 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Bootstrap starting.
I20260924 10:37:50.914767 18715 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Bootstrap starting.
I20260924 10:37:50.916399 18713 tablet_bootstrap.cc:654] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:50.916633 18715 tablet_bootstrap.cc:654] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:50.916855 18714 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Bootstrap starting.
I20260924 10:37:50.917387 18713 log.cc:824] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:50.917907 18715 log.cc:824] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:50.919145 18714 tablet_bootstrap.cc:654] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Neither blocks nor log segments found. Creating new log.
I20260924 10:37:50.919778 18715 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: No bootstrap required, opened a new log
I20260924 10:37:50.919802 18714 log.cc:824] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:37:50.919983 18715 ts_tablet_manager.cc:1397] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 10:37:50.921746 18713 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: No bootstrap required, opened a new log
I20260924 10:37:50.921950 18713 ts_tablet_manager.cc:1397] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 10:37:50.921963 18714 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: No bootstrap required, opened a new log
I20260924 10:37:50.922142 18714 ts_tablet_manager.cc:1397] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:37:50.924027 18715 raft_consensus.cc:348] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:50.924302 18715 raft_consensus.cc:374] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:50.924350 18715 raft_consensus.cc:729] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b8e3edd5164198921549580e7590d8, State: Initialized, Role: FOLLOWER
I20260924 10:37:50.924853 18715 consensus_queue.cc:260] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:50.924873 18713 raft_consensus.cc:348] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:50.925148 18713 raft_consensus.cc:374] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:50.925237 18713 raft_consensus.cc:729] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a933650b6eb3423686d3838d36c07b13, State: Initialized, Role: FOLLOWER
I20260924 10:37:50.925705 18713 consensus_queue.cc:260] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:50.925905 18428 heartbeater.cc:503] Master 127.2.193.190:34615 was elected leader, sending a full tablet report...
I20260924 10:37:50.925940 18715 ts_tablet_manager.cc:1428] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 10:37:50.925855 18714 raft_consensus.cc:348] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:50.926100 18714 raft_consensus.cc:374] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:37:50.926146 18714 raft_consensus.cc:729] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6186b1c0f4342fd8b3a7b109a4c443a, State: Initialized, Role: FOLLOWER
I20260924 10:37:50.926543 18714 consensus_queue.cc:260] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:50.926630 18713 ts_tablet_manager.cc:1428] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 10:37:50.926951 18696 heartbeater.cc:503] Master 127.2.193.190:34615 was elected leader, sending a full tablet report...
I20260924 10:37:50.927389 18714 ts_tablet_manager.cc:1428] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:37:50.927737 18562 heartbeater.cc:503] Master 127.2.193.190:34615 was elected leader, sending a full tablet report...
W20260924 10:37:51.103643 18697 tablet.cc:2369] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:37:51.137822 18563 tablet.cc:2369] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:37:51.165110 18429 tablet.cc:2369] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:37:51.245985 18720 raft_consensus.cc:482] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:37:51.246222 18720 raft_consensus.cc:504] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:51.247570 18720 leader_election.cc:288] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837), e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
I20260924 10:37:51.252619 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:51.248342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:49803 (local address 127.2.193.130:38257)
0924 10:37:51.248499 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:51.248505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:51.248752 (+ 247us) server_negotiation.cc:407] Connection header received
0924 10:37:51.248816 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:51.248819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:51.248870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:51.248965 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:51.249233 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.249899 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.251300 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.251913 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.252048 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:51.252104 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:51.252409 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 10:37:51.252482 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:37:51.252619 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:51.247975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:48491 (local address 127.2.193.129:38837)
0924 10:37:51.248119 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:51.248124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:51.248145 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:37:51.248211 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:51.248216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:51.248294 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:51.248399 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:37:51.248776 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.249636 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.251266 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.251921 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.252048 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:51.252104 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:51.252404 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 10:37:51.252482 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:37:51.252637 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:51.247860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38837 (local address 127.2.193.131:48491)
0924 10:37:51.247995 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:51.248012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:51.248116 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:51.248422 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:51.248426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:51.248447 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:51.248577 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:51.248585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.249776 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.249779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:51.251071 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:51.251082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.252065 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.252069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:51.252218 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:51.252286 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 10:37:51.252367 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:37:51.252458 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":52}
I20260924 10:37:51.252647 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:51.248175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:38257 (local address 127.2.193.131:49803)
0924 10:37:51.248630 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:51.248646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:51.248731 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:37:51.248987 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:37:51.248989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:37:51.249004 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:37:51.249092 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:51.249096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.250015 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.250018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:51.251177 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:37:51.251185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.252065 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.252069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:37:51.252206 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:51.252284 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 10:37:51.252367 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:37:51.252443 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":25,"thread_start_us":142,"threads_started":1}
I20260924 10:37:51.253458 18382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cfd035402d4b96a432d2b2408299ad" candidate_uuid: "a933650b6eb3423686d3838d36c07b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b8e3edd5164198921549580e7590d8" is_pre_election: true
I20260924 10:37:51.253783 18382 raft_consensus.cc:2454] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a933650b6eb3423686d3838d36c07b13 in term 0.
I20260924 10:37:51.253759 18516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cfd035402d4b96a432d2b2408299ad" candidate_uuid: "a933650b6eb3423686d3838d36c07b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" is_pre_election: true
I20260924 10:37:51.254043 18516 raft_consensus.cc:2454] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a933650b6eb3423686d3838d36c07b13 in term 0.
I20260924 10:37:51.254307 18584 leader_election.cc:302] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b8e3edd5164198921549580e7590d8, a933650b6eb3423686d3838d36c07b13; no voters:
I20260924 10:37:51.254586 18720 raft_consensus.cc:2781] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:37:51.254658 18720 raft_consensus.cc:482] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:37:51.254688 18720 raft_consensus.cc:3037] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:51.255806 18720 raft_consensus.cc:504] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:51.256174 18720 leader_election.cc:288] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [CANDIDATE]: Term 1 election: Requested vote from peers a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837), e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
I20260924 10:37:51.256573 18382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cfd035402d4b96a432d2b2408299ad" candidate_uuid: "a933650b6eb3423686d3838d36c07b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b8e3edd5164198921549580e7590d8"
I20260924 10:37:51.256708 18382 raft_consensus.cc:3037] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:51.256757 18516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00cfd035402d4b96a432d2b2408299ad" candidate_uuid: "a933650b6eb3423686d3838d36c07b13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a"
I20260924 10:37:51.256960 18516 raft_consensus.cc:3037] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:37:51.258165 18382 raft_consensus.cc:2454] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a933650b6eb3423686d3838d36c07b13 in term 1.
I20260924 10:37:51.258244 18516 raft_consensus.cc:2454] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a933650b6eb3423686d3838d36c07b13 in term 1.
I20260924 10:37:51.258522 18584 leader_election.cc:302] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b8e3edd5164198921549580e7590d8, a933650b6eb3423686d3838d36c07b13; no voters:
I20260924 10:37:51.258702 18720 raft_consensus.cc:2781] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:37:51.258924 18720 raft_consensus.cc:686] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 1 LEADER]: Becoming Leader. State: Replica: a933650b6eb3423686d3838d36c07b13, State: Running, Role: LEADER
I20260924 10:37:51.259320 18720 consensus_queue.cc:237] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:37:51.262625 18236 catalog_manager.cc:5495] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 reported cstate change: term changed from 0 to 1, leader changed from <none> to a933650b6eb3423686d3838d36c07b13 (127.2.193.131). New cstate: current_term: 1 leader_uuid: "a933650b6eb3423686d3838d36c07b13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } health_report { overall_health: HEALTHY } } }
I20260924 10:37:51.355068 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:37:51.351530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.2.193.131:42249)
0924 10:37:51.351668 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:37:51.351672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:37:51.351686 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:37:51.351780 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:37:51.351782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:37:51.351834 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:37:51.351938 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:37:51.352149 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.352647 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.353593 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:37:51.353810 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:37:51.353966 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:37:51.354212 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:37:51.354641 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:37:51.354730 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:37:51.354821 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:37:51.354872 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:37:51.361086 18382 raft_consensus.cc:1261] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Refusing update from remote peer a933650b6eb3423686d3838d36c07b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:51.361120 18516 raft_consensus.cc:1261] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 FOLLOWER]: Refusing update from remote peer a933650b6eb3423686d3838d36c07b13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:37:51.361929 18723 consensus_queue.cc:1035] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:51.362130 18720 consensus_queue.cc:1035] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:37:51.370458 18727 mvcc.cc:205] Tried to move back new op lower bound from 7332848727487692800 to 7332848727080742912. Current Snapshot: MvccSnapshot[applied={T|T < 7332848727487692800}]
I20260924 10:37:51.382133 18236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:51.382400 18236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:37:51.386281 18236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Sending DeleteTablet for 3 replicas of tablet 00cfd035402d4b96a432d2b2408299ad
I20260924 10:37:51.387532 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18299
I20260924 10:37:51.388356 18496 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cfd035402d4b96a432d2b2408299ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:51 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:41460
I20260924 10:37:51.388975 18736 tablet_replica.cc:330] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: stopping tablet replica
I20260924 10:37:51.388962 18630 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cfd035402d4b96a432d2b2408299ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:51 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:52378
I20260924 10:37:51.389712 18736 raft_consensus.cc:2229] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:37:51.390105 18736 raft_consensus.cc:2258] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:51.391728 18736 ts_tablet_manager.cc:1905] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:51.395588 18736 ts_tablet_manager.cc:1918] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:51.395704 18736 log.cc:1196] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/00cfd035402d4b96a432d2b2408299ad
I20260924 10:37:51.396055 18736 ts_tablet_manager.cc:1939] T 00cfd035402d4b96a432d2b2408299ad P e6186b1c0f4342fd8b3a7b109a4c443a: Deleting consensus metadata
W20260924 10:37:51.396975 18222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:38837 (error 108)
I20260924 10:37:51.397475 18222 catalog_manager.cc:4841] TS e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257): tablet 00cfd035402d4b96a432d2b2408299ad (table test-table [id=3248267ddbc7436b979317997f3707ad]) successfully deleted
W20260924 10:37:51.398175 18710 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:37:51.397829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38837 (local address 127.0.0.1:37216)
0924 10:37:51.397963 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 10:37:51.397980 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:37:51.398056 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38837: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:37:51.398307 18222 catalog_manager.cc:4584] TS a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00cfd035402d4b96a432d2b2408299ad: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38837: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260924 10:37:51.398608 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18433
I20260924 10:37:51.401432 18737 tablet_replica.cc:330] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: stopping tablet replica
I20260924 10:37:51.401847 18737 raft_consensus.cc:2229] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:37:51.402328 18737 raft_consensus.cc:2258] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:37:51.404060 18737 ts_tablet_manager.cc:1905] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:37:51.407681 18737 ts_tablet_manager.cc:1918] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:37:51.407775 18737 log.cc:1196] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/00cfd035402d4b96a432d2b2408299ad
I20260924 10:37:51.408115 18737 ts_tablet_manager.cc:1939] T 00cfd035402d4b96a432d2b2408299ad P a933650b6eb3423686d3838d36c07b13: Deleting consensus metadata
I20260924 10:37:51.408294 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18567
I20260924 10:37:51.415499 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18205
I20260924 10:38:11.424777 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34615
--webserver_interface=127.2.193.190
--webserver_port=34945
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:11.550709 18739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:11.551029 18739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:11.551141 18739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:11.555455 18739 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:38:11.555532 18739 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:38:11.555559 18739 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:38:11.555595 18739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:11.555639 18739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:11.555698 18739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:11.555735 18739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:11.555770 18739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:11.561424 18739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:11.562554 18739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:11.563675 18739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:11.577494 18745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:11.577606 18739 server_base.cc:1034] running on GCE node
W20260924 10:38:11.577435 18748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:11.577450 18746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:11.578433 18739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:11.579677 18739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:11.580848 18739 hybrid_clock.cc:648] HybridClock initialized: now 1790246291580828 us; error 34 us; skew 500 ppm
Sep 24 10:38:11 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:11.583752 18739 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:38:11.584695 18739 webserver.cc:460] Webserver started at http://127.2.193.190:34945/ using document root <none> and password file <none>
I20260924 10:38:11.585359 18739 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:11.585500 18739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:11.589516 18739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 10:38:11.591758 18755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:11.593088 18739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:38:11.593266 18739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "a9ec9b62965a4c71ab27fe700619256c"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "077b80fa86743ed91cfe13932062ea1d"
server_key_iv: "7f60a2e47f0697c6246da1be3c564cd1"
server_key_version: "encryptionkey@0"
I20260924 10:38:11.593915 18739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:11.614892 18739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:11.618409 18739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:11.618636 18739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:11.626006 18739 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:34615
I20260924 10:38:11.626338 18807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:34615 every 8 connection(s)
I20260924 10:38:11.627183 18739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:11.631803 18808 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:38:11.632124 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18739
I20260924 10:38:11.632553 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:38837
--local_ip_for_outbound_sockets=127.2.193.129
--tserver_master_addrs=127.2.193.190:34615
--webserver_port=44221
--webserver_interface=127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:38:11.633740 18808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Bootstrap starting.
I20260924 10:38:11.651170 18808 log.cc:824] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:11.659837 18808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:38:11.660264 18808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Bootstrap complete.
I20260924 10:38:11.664518 18808 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:11.664968 18808 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ec9b62965a4c71ab27fe700619256c, State: Initialized, Role: FOLLOWER
I20260924 10:38:11.665565 18808 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:11.665719 18808 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:11.665817 18808 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:11.665935 18808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:38:11.667928 18808 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:11.668296 18808 leader_election.cc:302] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9ec9b62965a4c71ab27fe700619256c; no voters:
I20260924 10:38:11.668653 18808 leader_election.cc:288] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:38:11.668737 18812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:38:11.668939 18812 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 LEADER]: Becoming Leader. State: Replica: a9ec9b62965a4c71ab27fe700619256c, State: Running, Role: LEADER
I20260924 10:38:11.669319 18812 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:11.669952 18808 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:11.670698 18813 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9ec9b62965a4c71ab27fe700619256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } } }
I20260924 10:38:11.670832 18813 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:11.671552 18814 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ec9b62965a4c71ab27fe700619256c. Latest consensus state: current_term: 2 leader_uuid: "a9ec9b62965a4c71ab27fe700619256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } } }
I20260924 10:38:11.671655 18814 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:11.672020 18818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:11.676792 18818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:11.677456 18818 catalog_manager.cc:1259] Loaded cluster ID: fe0d5dbf67624df8a016c9ed4e00f9d5
I20260924 10:38:11.677518 18818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:11.680027 18818 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:11.680730 18818 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Loaded TSK: 0
I20260924 10:38:11.698246 18818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Generated new TSK 1
I20260924 10:38:11.698503 18818 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:38:11.747503 18810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:11.747792 18810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:11.747864 18810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:11.751925 18810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:11.751979 18810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:11.752067 18810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:11.757151 18810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:38837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=44221
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:11.758212 18810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:11.759246 18810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:38:11.769970 18810 server_base.cc:1034] running on GCE node
W20260924 10:38:11.769917 18836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:11.770190 18838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:11.769901 18835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:11.770733 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:11.771516 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:11.772684 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790246291772664 us; error 55 us; skew 500 ppm
Sep 24 10:38:11 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:11.775794 18810 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:38:11.776671 18810 webserver.cc:460] Webserver started at http://127.2.193.129:44221/ using document root <none> and password file <none>
I20260924 10:38:11.777331 18810 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:11.777442 18810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:11.781569 18810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:11.783913 18845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:11.785413 18810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:11.785547 18810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "a1b8e3edd5164198921549580e7590d8"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "f114873d6aa04e4c5aa008ba471541e7"
server_key_iv: "63a1b6dd40bd24a14e112c043cb92125"
server_key_version: "encryptionkey@0"
I20260924 10:38:11.786103 18810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:11.815344 18810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:11.818766 18810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:11.819025 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:11.819742 18810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:11.820930 18852 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:38:11.825455 18810 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:38:11.825560 18810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 10:38:11.825663 18810 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:38:11.827461 18810 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:38:11.827539 18810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:38:11.827812 18852 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Bootstrap starting.
I20260924 10:38:11.838022 18810 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38837
I20260924 10:38:11.838105 18959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38837 every 8 connection(s)
I20260924 10:38:11.839030 18810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:11.839426 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18810
I20260924 10:38:11.839903 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:38257
--local_ip_for_outbound_sockets=127.2.193.130
--tserver_master_addrs=127.2.193.190:34615
--webserver_port=43911
--webserver_interface=127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:38:11.849730 18852 log.cc:824] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:38:11 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:11.856741 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:11.841147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:37843 (local address 127.2.193.190:34615)
0924 10:38:11.841775 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:11.841794 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:11.842411 (+ 617us) server_negotiation.cc:407] Connection header received
0924 10:38:11.843832 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:11.843844 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:11.844203 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:11.844554 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:11.845868 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.846846 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.848042 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.848525 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.848719 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:11.851570 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:11.851577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:11.851589 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:11.851619 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:11.853485 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:11.854050 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:11.854055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:11.854059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:11.854184 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:11.854534 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:11.854543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:11.854544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:11.854888 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:11.855033 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:11.855577 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 10:38:11.855959 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":111,"threads_started":1}
I20260924 10:38:11.856755 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:11.841170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:37843)
0924 10:38:11.842166 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:11.842238 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:11.843368 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:11.844601 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:11.844609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:11.845057 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:11.845684 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:11.845703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.847014 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.847017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:11.847912 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:11.847921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.848676 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.848681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:11.848870 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:11.849573 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:11.849589 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:11.851311 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:11.853625 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:11.853631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:11.853645 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:11.853915 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:11.854283 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:11.854287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:11.854289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:11.854430 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:11.855014 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:11.855020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:11.855267 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 10:38:11.855445 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 10:38:11.855791 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":129,"threads_started":1}
I20260924 10:38:11.859010 18852 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:38:11.859346 18960 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34615
I20260924 10:38:11.859550 18852 tablet_bootstrap.cc:492] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Bootstrap complete.
I20260924 10:38:11.859577 18960 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:11.860070 18960 heartbeater.cc:511] Master 127.2.193.190:34615 requested a full tablet report, sending...
I20260924 10:38:11.861003 18852 ts_tablet_manager.cc:1397] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.005s
I20260924 10:38:11.862370 18772 ts_manager.cc:194] Registered new tserver with Master: a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837)
I20260924 10:38:11.864044 18852 raft_consensus.cc:348] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:38:11.864658 18852 raft_consensus.cc:729] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b8e3edd5164198921549580e7590d8, State: Initialized, Role: FOLLOWER
I20260924 10:38:11.865234 18852 consensus_queue.cc:260] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } }
I20260924 10:38:11.865973 18772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:37843
I20260924 10:38:11.866360 18852 ts_tablet_manager.cc:1428] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:38:11.868873 18960 heartbeater.cc:503] Master 127.2.193.190:34615 was elected leader, sending a full tablet report...
I20260924 10:38:11.872370 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:11.866079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.2.193.129:38837)
0924 10:38:11.866541 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:11.866552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:11.867291 (+ 739us) server_negotiation.cc:407] Connection header received
0924 10:38:11.867348 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:11.867351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:11.867978 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:11.868140 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:11.868398 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.869241 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.871130 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.871733 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.871857 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:11.872006 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:11.872190 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:38:11.872264 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":96,"threads_started":1}
I20260924 10:38:11.872416 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:11.865284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38837 (local address 127.0.0.1:49804)
0924 10:38:11.867000 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:11.867047 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:11.867179 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:11.868130 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:11.868136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:11.868159 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:11.868286 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:11.868292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.869368 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.869373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:11.871008 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:11.871021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:11.871874 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:11.871878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:11.872001 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:11.872040 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:38:11.872152 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 10:38:11.872277 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"thread_start_us":143,"threads_started":1}
I20260924 10:38:11.873536 18894 tablet_service.cc:1514] Processing DeleteTablet for tablet 00cfd035402d4b96a432d2b2408299ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:37:51 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:49804
I20260924 10:38:11.874348 18969 tablet_replica.cc:330] stopping tablet replica
I20260924 10:38:11.874634 18969 raft_consensus.cc:2229] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:11.874872 18969 raft_consensus.cc:2258] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:11.875823 18969 ts_tablet_manager.cc:1905] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:11.879710 18969 ts_tablet_manager.cc:1918] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:11.879803 18969 log.cc:1196] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal/wals/00cfd035402d4b96a432d2b2408299ad
I20260924 10:38:11.880131 18969 ts_tablet_manager.cc:1939] T 00cfd035402d4b96a432d2b2408299ad P a1b8e3edd5164198921549580e7590d8: Deleting consensus metadata
I20260924 10:38:11.881973 18757 catalog_manager.cc:4841] TS a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837): tablet 00cfd035402d4b96a432d2b2408299ad (table test-table [id=3248267ddbc7436b979317997f3707ad]) successfully deleted
W20260924 10:38:11.964922 18962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:11.965260 18962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:11.965337 18962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:11.969542 18962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:11.969619 18962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:11.969703 18962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:11.975724 18962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:38257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=43911
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:11.976825 18962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:11.978004 18962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:11.987871 18975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:11.987854 18977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:11.988153 18962 server_base.cc:1034] running on GCE node
W20260924 10:38:11.987849 18974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:11.988605 18962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:11.989367 18962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:11.990547 18962 hybrid_clock.cc:648] HybridClock initialized: now 1790246291990502 us; error 56 us; skew 500 ppm
Sep 24 10:38:11 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:11.993438 18962 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:38:11.994339 18962 webserver.cc:460] Webserver started at http://127.2.193.130:43911/ using document root <none> and password file <none>
I20260924 10:38:11.994931 18962 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:11.995049 18962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:11.999205 18962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:38:12.001416 18984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:12.002715 18962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:38:12.002857 18962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "e6186b1c0f4342fd8b3a7b109a4c443a"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "21b3c880842e449ebb56f28db9600cc7"
server_key_iv: "7587c183783d13b68b534a9eba3998ed"
server_key_version: "encryptionkey@0"
I20260924 10:38:12.003435 18962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:12.017694 18962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:12.020788 18962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:12.021016 18962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:12.021682 18962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:12.022529 18962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:12.022603 18962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:12.022673 18962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:12.022727 18962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:12.031837 18962 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:38257
I20260924 10:38:12.031860 19097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:38257 every 8 connection(s)
I20260924 10:38:12.032883 18962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:12.039364 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 18962
I20260924 10:38:12.039860 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:42249
--local_ip_for_outbound_sockets=127.2.193.131
--tserver_master_addrs=127.2.193.190:34615
--webserver_port=46249
--webserver_interface=127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:38:12 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:12.048394 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.034776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:57269 (local address 127.2.193.190:34615)
0924 10:38:12.034962 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.034967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.035829 (+ 862us) server_negotiation.cc:407] Connection header received
0924 10:38:12.037139 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.037142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.037242 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.037350 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:12.038526 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.039050 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.040785 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.041037 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.041116 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.043924 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:12.043932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:12.043937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:12.043975 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:12.045907 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.046509 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.046515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.046519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.046623 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.046983 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.046988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.046990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.047267 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:12.047405 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.047970 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.048145 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:38:12.049116 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.035213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:57269)
0924 10:38:12.035664 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.035708 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:12.036914 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:12.037517 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:12.037524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:12.037884 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:12.038376 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.038390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.039186 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.039190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.040643 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.040658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.041232 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.041238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.041448 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.042078 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:12.042095 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:12.043708 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:12.046071 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:12.046077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:12.046091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:12.046394 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:12.046732 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:12.046735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:12.046736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:12.046881 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:12.047446 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:12.047451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:12.047708 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 10:38:12.047883 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 10:38:12.048170 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260924 10:38:12.050496 19098 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34615
I20260924 10:38:12.050793 19098 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:12.051486 19098 heartbeater.cc:511] Master 127.2.193.190:34615 requested a full tablet report, sending...
I20260924 10:38:12.052647 18772 ts_manager.cc:194] Registered new tserver with Master: e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
I20260924 10:38:12.053323 18772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:57269
W20260924 10:38:12.157007 19101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:12.157362 19101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:12.157415 19101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:12.161690 19101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:12.161756 19101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:12.161859 19101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:12.168071 19101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=46249
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:12.169256 19101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:12.170439 19101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:12.181051 19106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:12.181061 19107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:12.181075 19109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:12.181133 19101 server_base.cc:1034] running on GCE node
I20260924 10:38:12.181759 19101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:12.182489 19101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:12.183707 19101 hybrid_clock.cc:648] HybridClock initialized: now 1790246292183692 us; error 39 us; skew 500 ppm
Sep 24 10:38:12 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:12.186743 19101 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:38:12.187754 19101 webserver.cc:460] Webserver started at http://127.2.193.131:46249/ using document root <none> and password file <none>
I20260924 10:38:12.188337 19101 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:12.188444 19101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:12.192664 19101 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:38:12.195127 19116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:12.196552 19101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:38:12.196764 19101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "a933650b6eb3423686d3838d36c07b13"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "c7ef2bd91cd70b901c3926b6dff13c05"
server_key_iv: "5df72e4c18596e599a5ba8764c8bfca9"
server_key_version: "encryptionkey@0"
I20260924 10:38:12.197476 19101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:12.209824 19101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:12.213292 19101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:12.213505 19101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:12.214128 19101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:12.214977 19101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:12.215046 19101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:12.215119 19101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:12.215171 19101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:12.224237 19101 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:42249
I20260924 10:38:12.224259 19229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:42249 every 8 connection(s)
I20260924 10:38:12.225317 19101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:12.226935 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 19101
Sep 24 10:38:12 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:12.236611 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.227642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.2.193.190:34615)
0924 10:38:12.227846 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.227849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.228151 (+ 302us) server_negotiation.cc:407] Connection header received
0924 10:38:12.228302 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.228305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.228360 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.228481 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:12.229332 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.230157 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.231024 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.231349 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.231459 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.232949 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:12.232953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:12.232956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:12.232983 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:12.234767 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.235279 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.235282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.235283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.235365 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.235717 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.235722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.235726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.235952 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:12.236179 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.236284 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.236433 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260924 10:38:12.240250 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.227204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:35969 (local address 127.2.193.190:34615)
0924 10:38:12.227387 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.227394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.228480 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 10:38:12.230069 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.230071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.230109 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.230174 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:12.231467 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.232198 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.233726 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.233943 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.234059 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.236945 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:12.236949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:12.236952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:12.236974 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:12.238476 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.238957 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.238959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.238960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.239026 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.239314 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.239316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.239317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.239497 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:12.239571 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.240042 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.240135 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":67}
I20260924 10:38:12.241231 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.227756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:35969)
0924 10:38:12.228312 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.228369 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:12.229843 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:12.230368 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:12.230374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:12.230720 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:12.231308 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.231326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.232334 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.232338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.233584 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.233597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.234098 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.234103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.234285 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.234813 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:12.234856 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:12.236771 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:12.238598 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:12.238603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:12.238616 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:12.238869 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:12.239105 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:12.239107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:12.239109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:12.239238 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:12.239584 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:12.239589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:12.239822 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 10:38:12.239982 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 10:38:12.240237 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":150,"threads_started":1}
I20260924 10:38:12.242417 19230 heartbeater.cc:347] Connected to a master server at 127.2.193.190:34615
I20260924 10:38:12.242650 19230 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:12.243144 19230 heartbeater.cc:511] Master 127.2.193.190:34615 requested a full tablet report, sending...
I20260924 10:38:12.244390 18772 ts_manager.cc:194] Registered new tserver with Master: a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249)
I20260924 10:38:12.244963 18772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:35969
I20260924 10:38:12.250231 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:38:12.254808 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.250901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.2.193.190:34615)
0924 10:38:12.251053 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.251058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.251236 (+ 178us) server_negotiation.cc:407] Connection header received
0924 10:38:12.251473 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.251476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.251577 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.251692 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:12.251954 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.252481 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.253440 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.253695 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.253812 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.254137 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:12.254268 (+ 131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 10:38:12.254670 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43422: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 10:38:12.254895 18963 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43422: token expired
W20260924 10:38:12.255319 18699 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.193.190:34615)
I20260924 10:38:12.264623 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.255752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.2.193.190:34615)
0924 10:38:12.255878 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.255883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.256009 (+ 126us) server_negotiation.cc:407] Connection header received
0924 10:38:12.256157 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.256163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.256221 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.256315 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:12.257220 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.257939 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.259524 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.259729 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.259843 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.261525 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:12.261528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:12.261533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:12.261579 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:12.263201 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.263595 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.263598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.263599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.263672 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:12.263926 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:12.263928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:12.263929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:12.264127 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:12.264227 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.264411 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.264505 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:38:12.267197 18772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:38:12.270686 18772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:38:12.278883 18894 tablet_service.cc:1467] Processing CreateTablet for tablet d3a4428749d64207b073bdd06d1f66de (DEFAULT_TABLE table=test-table [id=6b491118795d455eb76fb22714aedcf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:12.279595 18894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a4428749d64207b073bdd06d1f66de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:12.282537 18852 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Bootstrap starting.
I20260924 10:38:12.282860 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.277435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:42249 (local address 127.0.0.1:57454)
0924 10:38:12.277556 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.277574 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:12.277688 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:12.278998 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:12.279001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:12.279019 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:12.279123 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.279129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.280497 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.280500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.281475 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.281494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.282479 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.282483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.282572 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.282601 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:38:12.282649 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 10:38:12.282726 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 10:38:12.283537 18852 tablet_bootstrap.cc:654] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:12.283674 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.277957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:38257 (local address 127.0.0.1:57400)
0924 10:38:12.278190 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.278201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:12.278300 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:12.279203 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:12.279207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:12.279229 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:12.279316 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.279324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.280805 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.280808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.281947 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.281958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.283279 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.283283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.283370 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.283393 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:38:12.283466 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:38:12.283535 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260924 10:38:12.283891 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.278151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.2.193.130:38257)
0924 10:38:12.278564 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.278573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.278590 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:38:12.278649 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.278653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.278840 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.279017 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:12.279451 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.280682 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.282081 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.282913 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.283046 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.283223 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.283523 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.283631 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":119,"threads_started":1}
I20260924 10:38:12.284137 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.277790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.2.193.131:42249)
0924 10:38:12.278451 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.278465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.278490 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:38:12.278576 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.278583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.278841 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.279016 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:12.279277 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.280358 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.281606 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.282325 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.283370 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.283544 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.283656 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.283924 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":132,"threads_started":1}
I20260924 10:38:12.285068 18852 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: No bootstrap required, opened a new log
I20260924 10:38:12.285163 18852 ts_tablet_manager.cc:1397] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 10:38:12.285614 18852 raft_consensus.cc:348] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.285750 18852 raft_consensus.cc:374] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:12.285785 18852 raft_consensus.cc:729] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b8e3edd5164198921549580e7590d8, State: Initialized, Role: FOLLOWER
I20260924 10:38:12.285904 18852 consensus_queue.cc:260] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.286161 18852 ts_tablet_manager.cc:1428] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:38:12.287114 19164 tablet_service.cc:1467] Processing CreateTablet for tablet d3a4428749d64207b073bdd06d1f66de (DEFAULT_TABLE table=test-table [id=6b491118795d455eb76fb22714aedcf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:12.287130 19032 tablet_service.cc:1467] Processing CreateTablet for tablet d3a4428749d64207b073bdd06d1f66de (DEFAULT_TABLE table=test-table [id=6b491118795d455eb76fb22714aedcf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:12.288034 19164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a4428749d64207b073bdd06d1f66de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:12.288200 19032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a4428749d64207b073bdd06d1f66de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:12.294663 19244 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Bootstrap starting.
I20260924 10:38:12.295284 19243 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Bootstrap starting.
I20260924 10:38:12.297611 19243 tablet_bootstrap.cc:654] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:12.298424 19243 log.cc:824] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:12.299492 19244 tablet_bootstrap.cc:654] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:12.300273 19244 log.cc:824] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:12.300310 19243 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: No bootstrap required, opened a new log
I20260924 10:38:12.300529 19243 ts_tablet_manager.cc:1397] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:38:12.302438 19244 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: No bootstrap required, opened a new log
I20260924 10:38:12.302628 19244 ts_tablet_manager.cc:1397] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260924 10:38:12.304665 19243 raft_consensus.cc:348] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.304957 19243 raft_consensus.cc:374] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:12.305007 19243 raft_consensus.cc:729] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a933650b6eb3423686d3838d36c07b13, State: Initialized, Role: FOLLOWER
I20260924 10:38:12.305502 19243 consensus_queue.cc:260] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.306216 19230 heartbeater.cc:503] Master 127.2.193.190:34615 was elected leader, sending a full tablet report...
I20260924 10:38:12.306551 19243 ts_tablet_manager.cc:1428] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:38:12.306432 19244 raft_consensus.cc:348] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.306720 19244 raft_consensus.cc:374] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:12.306780 19244 raft_consensus.cc:729] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6186b1c0f4342fd8b3a7b109a4c443a, State: Initialized, Role: FOLLOWER
I20260924 10:38:12.307245 19244 consensus_queue.cc:260] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.308085 19098 heartbeater.cc:503] Master 127.2.193.190:34615 was elected leader, sending a full tablet report...
I20260924 10:38:12.308444 19244 ts_tablet_manager.cc:1428] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260924 10:38:12.340752 18961 tablet.cc:2369] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:12.355742 19248 raft_consensus.cc:482] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:12.355942 19248 raft_consensus.cc:504] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.357051 19248 leader_election.cc:288] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249), a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837)
I20260924 10:38:12.361254 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.357285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:42249 (local address 127.2.193.130:35667)
0924 10:38:12.357440 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.357458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:12.357545 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:12.357958 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:12.357961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:12.357977 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:12.358069 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.358073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.359017 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.359020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.359969 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.359978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.360767 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.360770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.360879 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.360906 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:38:12.360988 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:38:12.361071 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":19}
I20260924 10:38:12.361307 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.357384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:35667 (local address 127.2.193.131:42249)
0924 10:38:12.357520 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.357525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.357543 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:38:12.357697 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.357703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.357762 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.357832 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:12.358178 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.358899 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.360111 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.360648 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.360744 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.360786 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.361041 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.361125 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:38:12.361514 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.357724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:54305 (local address 127.2.193.129:38837)
0924 10:38:12.357868 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.357873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.358043 (+ 170us) server_negotiation.cc:407] Connection header received
0924 10:38:12.358246 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.358248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.358290 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.358348 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:12.358643 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.359258 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.360401 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.360882 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.361019 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.361063 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.361319 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.361409 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:38:12.361752 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.357552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38837 (local address 127.2.193.130:54305)
0924 10:38:12.357944 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.357956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:12.358085 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:12.358383 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:12.358386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:12.358405 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:12.358529 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.358535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.359374 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.359377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.360260 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:12.360271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.361051 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.361054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:12.361153 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.361176 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:38:12.361285 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:38:12.361513 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1,"thread_start_us":148,"threads_started":1}
I20260924 10:38:12.362170 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4428749d64207b073bdd06d1f66de" candidate_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a933650b6eb3423686d3838d36c07b13" is_pre_election: true
I20260924 10:38:12.362269 18914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4428749d64207b073bdd06d1f66de" candidate_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b8e3edd5164198921549580e7590d8" is_pre_election: true
I20260924 10:38:12.362541 19184 raft_consensus.cc:2454] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6186b1c0f4342fd8b3a7b109a4c443a in term 0.
I20260924 10:38:12.362609 18914 raft_consensus.cc:2454] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6186b1c0f4342fd8b3a7b109a4c443a in term 0.
I20260924 10:38:12.363050 18986 leader_election.cc:302] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a933650b6eb3423686d3838d36c07b13, e6186b1c0f4342fd8b3a7b109a4c443a; no voters:
I20260924 10:38:12.363323 19248 raft_consensus.cc:2781] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:38:12.363437 19248 raft_consensus.cc:482] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:38:12.363500 19248 raft_consensus.cc:3037] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:12.364575 19248 raft_consensus.cc:504] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.364923 19248 leader_election.cc:288] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [CANDIDATE]: Term 1 election: Requested vote from peers a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249), a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837)
I20260924 10:38:12.365314 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4428749d64207b073bdd06d1f66de" candidate_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a933650b6eb3423686d3838d36c07b13"
I20260924 10:38:12.365418 18914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4428749d64207b073bdd06d1f66de" candidate_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1b8e3edd5164198921549580e7590d8"
I20260924 10:38:12.365456 19184 raft_consensus.cc:3037] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:12.365552 18914 raft_consensus.cc:3037] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:12.366645 18914 raft_consensus.cc:2454] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6186b1c0f4342fd8b3a7b109a4c443a in term 1.
I20260924 10:38:12.366652 19184 raft_consensus.cc:2454] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6186b1c0f4342fd8b3a7b109a4c443a in term 1.
I20260924 10:38:12.366938 18986 leader_election.cc:302] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1b8e3edd5164198921549580e7590d8, e6186b1c0f4342fd8b3a7b109a4c443a; no voters:
I20260924 10:38:12.367141 19248 raft_consensus.cc:2781] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:12.367434 19248 raft_consensus.cc:686] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 LEADER]: Becoming Leader. State: Replica: e6186b1c0f4342fd8b3a7b109a4c443a, State: Running, Role: LEADER
I20260924 10:38:12.367787 19248 consensus_queue.cc:237] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:12.371004 18772 catalog_manager.cc:5495] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a reported cstate change: term changed from 0 to 1, leader changed from <none> to e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130). New cstate: current_term: 1 leader_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } health_report { overall_health: UNKNOWN } } }
I20260924 10:38:12.382118 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.378793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.2.193.130:38257)
0924 10:38:12.378911 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:12.378915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:12.378927 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:38:12.379033 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:12.379035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:12.379085 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:12.379171 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:12.379360 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.379847 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.380683 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:12.380896 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:12.380981 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:12.381290 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:12.381688 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:38:12.381782 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:12.381947 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 10:38:12.381997 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:38:12.388235 19184 raft_consensus.cc:1261] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 1 FOLLOWER]: Refusing update from remote peer e6186b1c0f4342fd8b3a7b109a4c443a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:12.388257 18914 raft_consensus.cc:1261] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Refusing update from remote peer e6186b1c0f4342fd8b3a7b109a4c443a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:12.388985 19250 consensus_queue.cc:1035] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:12.389191 19248 consensus_queue.cc:1035] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:12.395555 19253 mvcc.cc:205] Tried to move back new op lower bound from 7332848813614350336 to 7332848813540941824. Current Snapshot: MvccSnapshot[applied={T|T < 7332848813614350336}]
I20260924 10:38:12.396848 19254 mvcc.cc:205] Tried to move back new op lower bound from 7332848813614350336 to 7332848813540941824. Current Snapshot: MvccSnapshot[applied={T|T < 7332848813614350336 or (T in {7332848813614350336})}]
I20260924 10:38:12.397275 19256 mvcc.cc:205] Tried to move back new op lower bound from 7332848813614350336 to 7332848813540941824. Current Snapshot: MvccSnapshot[applied={T|T < 7332848813614350336}]
I20260924 10:38:12.405085 18771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:12.405298 18771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:12.407753 18771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Sending DeleteTablet for 3 replicas of tablet d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:12.408566 19164 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a4428749d64207b073bdd06d1f66de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:12 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:57454
I20260924 10:38:12.408624 19032 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a4428749d64207b073bdd06d1f66de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:12 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:57400
I20260924 10:38:12.409034 18894 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a4428749d64207b073bdd06d1f66de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:12 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:49804
I20260924 10:38:12.409128 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18810
I20260924 10:38:12.409366 19263 tablet_replica.cc:330] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: stopping tablet replica
I20260924 10:38:12.409096 19262 tablet_replica.cc:330] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: stopping tablet replica
I20260924 10:38:12.409830 19263 raft_consensus.cc:2229] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:38:12.410260 19263 raft_consensus.cc:2258] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:12.410725 19262 raft_consensus.cc:2229] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:12.411033 19262 raft_consensus.cc:2258] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:12.412604 19262 ts_tablet_manager.cc:1905] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:12.415422 19262 ts_tablet_manager.cc:1918] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:12.415540 19262 log.cc:1196] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:12.415876 19262 ts_tablet_manager.cc:1939] T d3a4428749d64207b073bdd06d1f66de P a933650b6eb3423686d3838d36c07b13: Deleting consensus metadata
I20260924 10:38:12.416818 19263 ts_tablet_manager.cc:1905] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:38:12.416935 18757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:38837 (error 108)
I20260924 10:38:12.417831 18757 catalog_manager.cc:4841] TS a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249): tablet d3a4428749d64207b073bdd06d1f66de (table test-table [id=6b491118795d455eb76fb22714aedcf8]) successfully deleted
W20260924 10:38:12.418231 19240 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:38:12.417691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:38837 (local address 127.0.0.1:49816)
0924 10:38:12.417836 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.417853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:12.417948 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:12.418139 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 10:38:12.418404 18757 catalog_manager.cc:4584] TS a1b8e3edd5164198921549580e7590d8 (127.2.193.129:38837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3a4428749d64207b073bdd06d1f66de: Network error: Client connection negotiation failed: client connection to 127.2.193.129:38837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:38:12.419528 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18962
I20260924 10:38:12.425356 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 19101
I20260924 10:38:12.427131 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:12.425651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:38257 (local address 127.0.0.1:57422)
0924 10:38:12.426943 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:12.427045 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1198}
I20260924 10:38:12.431070 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 18739
I20260924 10:38:32.437664 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34615
--webserver_interface=127.2.193.190
--webserver_port=34945
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:32.550964 19265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:32.551273 19265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:32.551316 19265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:32.555382 19265 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:38:32.555439 19265 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:38:32.555455 19265 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:38:32.555469 19265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:32.555488 19265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:32.555506 19265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:32.555522 19265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:32.555543 19265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:32.561008 19265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:34615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=34945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:32.562109 19265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:32.563126 19265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:32.572046 19273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:32.572041 19270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:32.572103 19265 server_base.cc:1034] running on GCE node
W20260924 10:38:32.572041 19271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:32.572943 19265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:32.574303 19265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:32.575503 19265 hybrid_clock.cc:648] HybridClock initialized: now 1790246312575484 us; error 31 us; skew 500 ppm
Sep 24 10:38:32 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:32.578478 19265 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:38:32.579399 19265 webserver.cc:460] Webserver started at http://127.2.193.190:34945/ using document root <none> and password file <none>
I20260924 10:38:32.579982 19265 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:32.580071 19265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:32.584136 19265 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:38:32.586385 19280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:32.587664 19265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:38:32.587806 19265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "a9ec9b62965a4c71ab27fe700619256c"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "077b80fa86743ed91cfe13932062ea1d"
server_key_iv: "7f60a2e47f0697c6246da1be3c564cd1"
server_key_version: "encryptionkey@0"
I20260924 10:38:32.588369 19265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:32.615190 19265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:32.618975 19265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:32.619194 19265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:32.626333 19265 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:34615
I20260924 10:38:32.626650 19332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:34615 every 8 connection(s)
I20260924 10:38:32.627518 19265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:32.631912 19333 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:38:32.633580 19333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Bootstrap starting.
I20260924 10:38:32.634706 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 19265
I20260924 10:38:32.635138 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:38837
--local_ip_for_outbound_sockets=127.2.193.129
--tserver_master_addrs=127.2.193.190:34615
--webserver_port=44221
--webserver_interface=127.2.193.129
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:38:32.651085 19333 log.cc:824] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:32.662371 19333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:38:32.662798 19333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Bootstrap complete.
I20260924 10:38:32.666888 19333 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:32.667286 19333 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ec9b62965a4c71ab27fe700619256c, State: Initialized, Role: FOLLOWER
I20260924 10:38:32.667858 19333 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:32.667979 19333 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:32.668075 19333 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:32.668169 19333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:38:32.670123 19333 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:32.670454 19333 leader_election.cc:302] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9ec9b62965a4c71ab27fe700619256c; no voters:
I20260924 10:38:32.670785 19333 leader_election.cc:288] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:38:32.670866 19337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:38:32.671051 19337 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [term 3 LEADER]: Becoming Leader. State: Replica: a9ec9b62965a4c71ab27fe700619256c, State: Running, Role: LEADER
I20260924 10:38:32.671392 19337 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } }
I20260924 10:38:32.672017 19333 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:32.672413 19337 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ec9b62965a4c71ab27fe700619256c. Latest consensus state: current_term: 3 leader_uuid: "a9ec9b62965a4c71ab27fe700619256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } } }
I20260924 10:38:32.672622 19337 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:32.672791 19338 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a9ec9b62965a4c71ab27fe700619256c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ec9b62965a4c71ab27fe700619256c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 34615 } } }
I20260924 10:38:32.672967 19338 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:32.673031 19341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:32.678323 19341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:32.679086 19341 catalog_manager.cc:1259] Loaded cluster ID: fe0d5dbf67624df8a016c9ed4e00f9d5
I20260924 10:38:32.679152 19341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:32.681730 19341 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:32.682475 19341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Loaded TSK: 0
I20260924 10:38:32.682569 19341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9ec9b62965a4c71ab27fe700619256c: Loaded TSK: 1
F20260924 10:38:32.683323 19341 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f775b85a9ad google::LogMessage::Fail() at ??:0
@ 0x7f775b85d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f775b85a4de google::LogMessage::Flush() at ??:0
@ 0x7f775b85ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f775e51e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f775e516f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f77661bf7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f776619999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f776619a178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f77661dbdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f77661fa82b std::function<>::operator()() at ??:0
@ 0x7f7766199b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f776619af3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7766197579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f77661db640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f775daac3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f775daacc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f775daae3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f775da9d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f775f0e16db start_thread at ??:0
@ 0x7f775a35861f clone at ??:0
W20260924 10:38:32.775501 19335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:32.775787 19335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:32.775893 19335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:32.780051 19335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:32.780134 19335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:32.780231 19335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:32.785830 19335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:38837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=44221
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:32.787061 19335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:32.788296 19335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:32.798471 19381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:32.798466 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:32.798471 19380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:32.799009 19335 server_base.cc:1034] running on GCE node
I20260924 10:38:32.799418 19335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:32.800069 19335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:32.801281 19335 hybrid_clock.cc:648] HybridClock initialized: now 1790246312801249 us; error 46 us; skew 500 ppm
Sep 24 10:38:32 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:32.804054 19335 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:38:32.805032 19335 webserver.cc:460] Webserver started at http://127.2.193.129:44221/ using document root <none> and password file <none>
I20260924 10:38:32.805634 19335 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:32.805697 19335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:32.809703 19335 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:38:32.812004 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:32.813459 19335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:32.813577 19335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "a1b8e3edd5164198921549580e7590d8"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "f114873d6aa04e4c5aa008ba471541e7"
server_key_iv: "63a1b6dd40bd24a14e112c043cb92125"
server_key_version: "encryptionkey@0"
I20260924 10:38:32.814146 19335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:32.825908 19335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:32.829409 19335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:32.829620 19335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:32.830246 19335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:32.831359 19397 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:38:32.836921 19335 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:38:32.837039 19335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 10:38:32.837145 19335 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:38:32.838894 19335 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:38:32.838981 19335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:38:32.839295 19397 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Bootstrap starting.
I20260924 10:38:32.849910 19335 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:38837
I20260924 10:38:32.849949 19504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:38837 every 8 connection(s)
I20260924 10:38:32.850878 19335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:32.852046 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 19335
I20260924 10:38:32.852490 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:38257
--local_ip_for_outbound_sockets=127.2.193.130
--tserver_master_addrs=127.2.193.190:34615
--webserver_port=43911
--webserver_interface=127.2.193.130
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:38:32.861223 19397 log.cc:824] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:32.870692 19397 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:38:32.871167 19397 tablet_bootstrap.cc:492] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Bootstrap complete.
I20260924 10:38:32.872620 19397 ts_tablet_manager.cc:1397] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.012s
I20260924 10:38:32.876077 19397 raft_consensus.cc:348] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:32.876646 19397 raft_consensus.cc:729] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b8e3edd5164198921549580e7590d8, State: Initialized, Role: FOLLOWER
I20260924 10:38:32.877252 19397 consensus_queue.cc:260] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:32.878369 19397 ts_tablet_manager.cc:1428] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260924 10:38:32.927047 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:38:32.853116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:58357)
0924 10:38:32.854473 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:32.854540 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:32.855735 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:32.926059 (+ 70324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":136,"threads_started":1}
W20260924 10:38:32.927627 19505 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:34615 (0 consecutive failures): Network error: Failed to ping master at 127.2.193.190:34615: Client connection negotiation failed: client connection to 127.2.193.190:34615: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:38:32.928572 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:32.928223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:51193)
0924 10:38:32.928390 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:32.928472 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 10:38:32.929371 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:32.929148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:45129)
0924 10:38:32.929266 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:32.929311 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 10:38:32.929605 19505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:38:32.974870 19508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:32.975100 19508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:32.975163 19508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:32.979101 19508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:32.979151 19508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:32.979207 19508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:32.984239 19508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:38257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=43911
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:32.985275 19508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:32.986227 19508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:32.995601 19515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:32.995601 19516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:32.995601 19518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:32.996411 19508 server_base.cc:1034] running on GCE node
I20260924 10:38:32.996848 19508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:32.997502 19508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:32.998677 19508 hybrid_clock.cc:648] HybridClock initialized: now 1790246312998659 us; error 28 us; skew 500 ppm
Sep 24 10:38:32 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:33.001367 19508 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:38:33.002242 19508 webserver.cc:460] Webserver started at http://127.2.193.130:43911/ using document root <none> and password file <none>
I20260924 10:38:33.002799 19508 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:33.002885 19508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:33.006714 19508 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:38:33.008759 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:33.009970 19508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:38:33.010094 19508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "e6186b1c0f4342fd8b3a7b109a4c443a"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "21b3c880842e449ebb56f28db9600cc7"
server_key_iv: "7587c183783d13b68b534a9eba3998ed"
server_key_version: "encryptionkey@0"
I20260924 10:38:33.010641 19508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:33.022069 19508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:33.025041 19508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:33.025318 19508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:33.025853 19508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:33.026811 19532 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 10:38:33.030369 19532 ts_tablet_manager.cc:1852] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 10:38:33.030745 19532 ts_tablet_manager.cc:1905] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:33.034663 19532 ts_tablet_manager.cc:1918] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:33.034750 19532 log.cc:1196] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:33.035133 19532 ts_tablet_manager.cc:1939] T d3a4428749d64207b073bdd06d1f66de P e6186b1c0f4342fd8b3a7b109a4c443a: Deleting consensus metadata
I20260924 10:38:33.035429 19508 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 10:38:33.035511 19508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260924 10:38:33.035609 19508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:33.035667 19508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:33.044617 19508 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:38257
I20260924 10:38:33.044656 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:38257 every 8 connection(s)
I20260924 10:38:33.045527 19508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:33.046023 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 19508
I20260924 10:38:33.046522 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:42249
--local_ip_for_outbound_sockets=127.2.193.131
--tserver_master_addrs=127.2.193.190:34615
--webserver_port=46249
--webserver_interface=127.2.193.131
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:38:33.049467 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:33.048065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:55125)
0924 10:38:33.048539 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:33.048666 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
W20260924 10:38:33.049901 19640 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:34615 (0 consecutive failures): Network error: Failed to ping master at 127.2.193.190:34615: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
I20260924 10:38:33.050585 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:33.050368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:38573)
0924 10:38:33.050473 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:33.050523 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 10:38:33.051275 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:33.051104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:34591)
0924 10:38:33.051196 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:33.051233 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:38:33.051491 19640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:38:33.154929 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:33.155190 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:33.155233 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:33.159539 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:33.159613 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:33.159715 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:33.165232 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=46249
--tserver_master_addrs=127.2.193.190:34615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:33.166309 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:33.167359 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:33.177049 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:33.177043 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:33.177050 19642 server_base.cc:1034] running on GCE node
W20260924 10:38:33.177084 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:33.177749 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:33.178350 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:33.179513 19642 hybrid_clock.cc:648] HybridClock initialized: now 1790246313179473 us; error 53 us; skew 500 ppm
Sep 24 10:38:33 dist-test-slave-8wc7 krb5kdc[18192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:33.182387 19642 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:38:33.183306 19642 webserver.cc:460] Webserver started at http://127.2.193.131:46249/ using document root <none> and password file <none>
I20260924 10:38:33.183867 19642 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:33.183941 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:33.187896 19642 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:38:33.190088 19658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:33.191345 19642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 10:38:33.191488 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "a933650b6eb3423686d3838d36c07b13"
format_stamp: "Formatted at 2026-09-24 10:37:50 on dist-test-slave-8wc7"
server_key: "c7ef2bd91cd70b901c3926b6dff13c05"
server_key_iv: "5df72e4c18596e599a5ba8764c8bfca9"
server_key_version: "encryptionkey@0"
I20260924 10:38:33.192071 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:33.208635 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:33.211645 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:33.211867 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:33.212463 19642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:33.213356 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:33.213423 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:33.213490 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:33.213538 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:33.223058 19642 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:42249
I20260924 10:38:33.223098 19771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:42249 every 8 connection(s)
I20260924 10:38:33.224084 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:33.228086 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:33.226370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:45045)
0924 10:38:33.226810 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:33.226917 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
W20260924 10:38:33.228626 19772 heartbeater.cc:644] Failed to heartbeat to 127.2.193.190:34615 (0 consecutive failures): Network error: Failed to ping master at 127.2.193.190:34615: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
I20260924 10:38:33.229283 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:33.229014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:36857)
0924 10:38:33.229138 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:33.229221 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 10:38:33.229907 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:33.229741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:45903)
0924 10:38:33.229827 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:33.229867 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 10:38:33.230237 19772 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:38:33.233326 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 19642
I20260924 10:38:33.930542 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:33.930141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:58455)
0924 10:38:33.930375 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:33.930426 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260924 10:38:34.052451 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:34.052039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:55469)
0924 10:38:34.052308 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:34.052364 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260924 10:38:34.231289 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:34.230879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:35811)
0924 10:38:34.231131 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:34.231188 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260924 10:38:34.607268 19779 raft_consensus.cc:482] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:34.607510 19779 raft_consensus.cc:504] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:34.608592 19779 leader_election.cc:288] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249), e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
Sep 24 10:38:34 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.130@KRBTEST.COM
Sep 24 10:38:34 dist-test-slave-8wc7 krb5kdc[18192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.131@KRBTEST.COM
I20260924 10:38:34.623104 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:34.608978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:41819 (local address 127.2.193.131:42249)
0924 10:38:34.609362 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:34.609425 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:34.609542 (+ 117us) server_negotiation.cc:407] Connection header received
0924 10:38:34.610350 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:34.610356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:34.610707 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:34.611001 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:34.612228 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.613240 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.614474 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.615014 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.615167 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:34.618488 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:34.618498 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:34.618517 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:34.618559 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:34.620384 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:34.620912 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:34.620918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:34.620923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:34.621088 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:34.621545 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:34.621549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:34.621553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:34.622032 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:34.622161 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:34.622682 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 10:38:34.622937 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":129,"threads_started":1}
I20260924 10:38:34.623095 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:34.609206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:38257 (local address 127.2.193.129:48255)
0924 10:38:34.609486 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:34.609498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:34.609565 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:34.611077 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:34.611083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:34.611605 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:34.612053 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:34.612068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.613379 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.613382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:34.614353 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:34.614362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.615207 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.615211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:34.615334 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:34.616054 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:34.616088 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:34.617889 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:34.620361 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:34.620366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:34.620377 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:34.620790 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:34.621203 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:34.621206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:34.621209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:34.621500 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:34.622163 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:34.622167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:34.622381 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 10:38:34.622557 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 10:38:34.622892 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":191,"spinlock_wait_cycles":41600,"thread_start_us":61,"threads_started":1}
I20260924 10:38:34.623158 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:34.609398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:48255 (local address 127.2.193.130:38257)
0924 10:38:34.609733 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:34.609744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:34.609774 (+ 30us) server_negotiation.cc:407] Connection header received
0924 10:38:34.610411 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:34.610417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:34.610707 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:34.611028 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:34.612233 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.613228 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.614476 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.615087 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.615253 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:34.618141 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:34.618146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:34.618157 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:34.618184 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:34.620243 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:34.620920 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:34.620925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:34.620930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:34.621088 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:34.621590 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:34.621593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:34.621594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:34.622020 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:34.622179 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:34.622688 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 10:38:34.622956 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260924 10:38:34.623142 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:34.608817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:42249 (local address 127.2.193.129:41819)
0924 10:38:34.609420 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:34.609435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:34.609564 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:34.611075 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:34.611081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:34.611598 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:34.612053 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:34.612069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.613376 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.613381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:34.614352 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:34.614362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:34.615137 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:34.615144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:34.615344 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:34.616118 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:34.616133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:34.618239 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:34.620498 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:34.620501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:34.620503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:34.620790 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:34.621203 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:34.621207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:34.621209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:34.621434 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:34.622139 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:34.622146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:34.622381 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 10:38:34.622557 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 10:38:34.622974 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":228,"spinlock_wait_cycles":39296,"thread_start_us":88,"threads_started":1}
W20260924 10:38:34.625703 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
W20260924 10:38:34.626302 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:34.626441 19392 leader_election.cc:302] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b8e3edd5164198921549580e7590d8; no voters: a933650b6eb3423686d3838d36c07b13, e6186b1c0f4342fd8b3a7b109a4c443a
I20260924 10:38:34.626665 19779 raft_consensus.cc:2726] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:38:34.931831 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:34.931476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:33327)
0924 10:38:34.931633 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:34.931714 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 10:38:35.053603 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:35.053152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:57325)
0924 10:38:35.053447 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:35.053502 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":72,"threads_started":1}
I20260924 10:38:35.232533 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:35.232103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:47009)
0924 10:38:35.232371 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:35.232427 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260924 10:38:35.933045 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:35.932567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:35047)
0924 10:38:35.932862 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:35.932924 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260924 10:38:36.054860 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:36.054379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:46579)
0924 10:38:36.054676 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:36.054741 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260924 10:38:36.233865 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:36.233388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:42821)
0924 10:38:36.233690 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:36.233751 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260924 10:38:36.831189 19789 raft_consensus.cc:482] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:36.831323 19789 raft_consensus.cc:504] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:36.831703 19789 leader_election.cc:288] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249), e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
W20260924 10:38:36.832715 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
W20260924 10:38:36.832876 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:36.832947 19392 leader_election.cc:302] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b8e3edd5164198921549580e7590d8; no voters: a933650b6eb3423686d3838d36c07b13, e6186b1c0f4342fd8b3a7b109a4c443a
I20260924 10:38:36.833148 19789 raft_consensus.cc:2726] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:38:36.934303 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:36.933882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:37125)
0924 10:38:36.934137 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:36.934206 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
I20260924 10:38:37.056042 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:37.055601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:46551)
0924 10:38:37.055882 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:37.055937 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
I20260924 10:38:37.234997 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:37.234551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:45937)
0924 10:38:37.234831 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:37.234888 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260924 10:38:37.935858 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:37.935311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:40693)
0924 10:38:37.935685 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:37.935754 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":95,"threads_started":1}
I20260924 10:38:38.057350 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:38.056845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:59277)
0924 10:38:38.057126 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:38.057252 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260924 10:38:38.236377 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:38.235945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:34227)
0924 10:38:38.236216 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:38.236279 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260924 10:38:38.937400 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:38.936888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:35723)
0924 10:38:38.937159 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:38.937297 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260924 10:38:39.058772 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:39.058289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:59053)
0924 10:38:39.058602 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:39.058669 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
I20260924 10:38:39.237754 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:39.237291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:49451)
0924 10:38:39.237547 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:39.237641 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260924 10:38:39.849555 19799 raft_consensus.cc:482] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:39.849694 19799 raft_consensus.cc:504] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:39.850092 19799 leader_election.cc:288] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249), e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
W20260924 10:38:39.851068 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
W20260924 10:38:39.851217 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:39.851269 19392 leader_election.cc:302] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b8e3edd5164198921549580e7590d8; no voters: a933650b6eb3423686d3838d36c07b13, e6186b1c0f4342fd8b3a7b109a4c443a
I20260924 10:38:39.851415 19799 raft_consensus.cc:2726] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:38:39.938797 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:39.938305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:60133)
0924 10:38:39.938622 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:39.938691 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260924 10:38:40.060102 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:40.059616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:43367)
0924 10:38:40.059942 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:40.060002 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260924 10:38:40.239054 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:40.238578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:59673)
0924 10:38:40.238891 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:40.238949 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":73,"threads_started":1}
I20260924 10:38:40.940099 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:40.939635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:36863)
0924 10:38:40.939925 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:40.939991 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260924 10:38:41.061555 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:41.061039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:34563)
0924 10:38:41.061371 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:41.061455 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":78,"threads_started":1}
I20260924 10:38:41.240496 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:41.240067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:54639)
0924 10:38:41.240334 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:41.240394 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260924 10:38:41.941330 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:41.940870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:55403)
0924 10:38:41.941120 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:41.941229 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260924 10:38:41.954380 19807 raft_consensus.cc:482] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:41.954535 19807 raft_consensus.cc:504] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:41.955008 19807 leader_election.cc:288] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249), e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
W20260924 10:38:41.955798 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
W20260924 10:38:41.955960 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:41.956024 19392 leader_election.cc:302] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b8e3edd5164198921549580e7590d8; no voters: a933650b6eb3423686d3838d36c07b13, e6186b1c0f4342fd8b3a7b109a4c443a
I20260924 10:38:41.956202 19807 raft_consensus.cc:2726] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:38:42.062906 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:42.062433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:44413)
0924 10:38:42.062741 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:42.062805 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":82,"threads_started":1}
I20260924 10:38:42.242046 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:42.241434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:46163)
0924 10:38:42.241779 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:42.241867 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260924 10:38:42.942728 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:42.942253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:51899)
0924 10:38:42.942561 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:42.942626 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260924 10:38:43.064329 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:43.063834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:59159)
0924 10:38:43.064152 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:43.064222 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":85,"threads_started":1}
I20260924 10:38:43.243381 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:43.242934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:33109)
0924 10:38:43.243209 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:43.243271 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260924 10:38:43.944011 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:43.943594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:37369)
0924 10:38:43.943850 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:43.943911 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260924 10:38:44.065658 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:44.065239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:54473)
0924 10:38:44.065490 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:44.065548 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
I20260924 10:38:44.244917 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:44.244298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:49301)
0924 10:38:44.244654 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:44.244754 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260924 10:38:44.945379 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:44.944835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:50313)
0924 10:38:44.945096 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:44.945156 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260924 10:38:45.066929 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:45.066485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:41033)
0924 10:38:45.066764 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:45.066823 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260924 10:38:45.246043 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:45.245581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:42487)
0924 10:38:45.245838 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:45.245893 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260924 10:38:45.407037 19819 raft_consensus.cc:482] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:45.407161 19819 raft_consensus.cc:504] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a933650b6eb3423686d3838d36c07b13" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 42249 } } peers { permanent_uuid: "e6186b1c0f4342fd8b3a7b109a4c443a" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 38257 } } peers { permanent_uuid: "a1b8e3edd5164198921549580e7590d8" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 38837 } }
I20260924 10:38:45.407512 19819 leader_election.cc:288] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249), e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257)
W20260924 10:38:45.408288 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a933650b6eb3423686d3838d36c07b13 (127.2.193.131:42249): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
W20260924 10:38:45.408458 19392 leader_election.cc:341] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6186b1c0f4342fd8b3a7b109a4c443a (127.2.193.130:38257): Not found: Tablet not found: d3a4428749d64207b073bdd06d1f66de
I20260924 10:38:45.408546 19392 leader_election.cc:302] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1b8e3edd5164198921549580e7590d8; no voters: a933650b6eb3423686d3838d36c07b13, e6186b1c0f4342fd8b3a7b109a4c443a
I20260924 10:38:45.408689 19819 raft_consensus.cc:2726] T d3a4428749d64207b073bdd06d1f66de P a1b8e3edd5164198921549580e7590d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:38:45.946565 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:45.946148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:45363)
0924 10:38:45.946396 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:45.946451 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260924 10:38:46.068066 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:46.067673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:56797)
0924 10:38:46.067907 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:46.067962 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260924 10:38:46.247627 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:46.247038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:44647)
0924 10:38:46.247412 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:46.247487 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260924 10:38:46.947901 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:46.947362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:53525)
0924 10:38:46.947703 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:46.947769 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":89,"threads_started":1}
I20260924 10:38:47.069350 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:47.068848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:49585)
0924 10:38:47.069113 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:47.069250 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260924 10:38:47.249043 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:47.248560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:48745)
0924 10:38:47.248866 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:47.248934 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260924 10:38:47.949370 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:47.948900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.129:35665)
0924 10:38:47.949155 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:47.949276 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260924 10:38:48.070585 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:48.070188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.130:57153)
0924 10:38:48.070433 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:48.070483 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260924 10:38:48.238822 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 19335
I20260924 10:38:48.246241 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 19508
I20260924 10:38:48.251240 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:48.250177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:34615 (local address 127.2.193.131:37485)
0924 10:38:48.251036 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:48.251110 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.190:34615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":79,"threads_started":1}
I20260924 10:38:48.252944 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 19642
2026-09-24T10:38:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59487 ms)
[----------] 30 tests from SecurityITest (242351 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:38:48 dist-test-slave-8wc7 krb5kdc[19831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:38:48 dist-test-slave-8wc7 krb5kdc[19831](info): set up 2 sockets
Sep 24 10:38:48 dist-test-slave-8wc7 krb5kdc[19831](info): commencing operation
krb5kdc: starting...
W20260924 10:38:49.309113 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:38:49 dist-test-slave-8wc7 krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:38:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:38:49.437400 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:38669
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:49.550339 19844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:49.550647 19844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:49.550690 19844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:49.554723 19844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:49.554790 19844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:49.554811 19844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:49.554827 19844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:49.554842 19844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:49.560357 19844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:38669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:49.561461 19844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:49.562472 19844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:49.571121 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:49.571122 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:49.571122 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:49.571646 19844 server_base.cc:1034] running on GCE node
I20260924 10:38:49.572167 19844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:49.573354 19844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:49.574549 19844 hybrid_clock.cc:648] HybridClock initialized: now 1790246329574520 us; error 52 us; skew 500 ppm
Sep 24 10:38:49 dist-test-slave-8wc7 krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246329, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:49.577483 19844 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:38:49.578411 19844 webserver.cc:460] Webserver started at http://127.2.193.190:45811/ using document root <none> and password file <none>
I20260924 10:38:49.578987 19844 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:49.579073 19844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:49.579298 19844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:49.581511 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "49d94d49a1fd4ef8855df959c74b2575"
format_stamp: "Formatted at 2026-09-24 10:38:49 on dist-test-slave-8wc7"
server_key: "e6e026a88c595b13da6496a251106100"
server_key_iv: "7fe70e2c4acacb6ea6cdac83081056f6"
server_key_version: "encryptionkey@0"
I20260924 10:38:49.582096 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "49d94d49a1fd4ef8855df959c74b2575"
format_stamp: "Formatted at 2026-09-24 10:38:49 on dist-test-slave-8wc7"
server_key: "e6e026a88c595b13da6496a251106100"
server_key_iv: "7fe70e2c4acacb6ea6cdac83081056f6"
server_key_version: "encryptionkey@0"
I20260924 10:38:49.585960 19844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:38:49.588553 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:49.589890 19844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:49.590034 19844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "49d94d49a1fd4ef8855df959c74b2575"
format_stamp: "Formatted at 2026-09-24 10:38:49 on dist-test-slave-8wc7"
server_key: "e6e026a88c595b13da6496a251106100"
server_key_iv: "7fe70e2c4acacb6ea6cdac83081056f6"
server_key_version: "encryptionkey@0"
I20260924 10:38:49.590153 19844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:49.609086 19844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:49.612677 19844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:49.612900 19844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:49.620307 19844 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:38669
I20260924 10:38:49.620332 19911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:38669 every 8 connection(s)
I20260924 10:38:49.621340 19844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:49.623987 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 19844
I20260924 10:38:49.624120 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:38:49.624385 19912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:49.624428 2822 external_mini_cluster.cc:1428] Setting key ccca0c82a6737139f04ebc887b3a4b2a
I20260924 10:38:49.631414 19912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575: Bootstrap starting.
Sep 24 10:38:49 dist-test-slave-8wc7 krb5kdc[19831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:49.634439 19912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:49.635201 19912 log.cc:824] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:49.637601 19912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575: No bootstrap required, opened a new log
I20260924 10:38:49.639947 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:49.625932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.2.193.190:38669)
0924 10:38:49.626357 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:49.626376 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:49.626411 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:38:49.627018 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:49.627034 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:49.627362 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:49.627695 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:49.628641 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:49.629695 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:49.630897 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:49.631382 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:49.631568 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:49.634471 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:49.634477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:49.634522 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:49.634563 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:49.636684 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:49.637162 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:49.637168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:49.637201 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:49.637345 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:49.637610 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:49.637616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:49.637620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:49.638138 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:49.638329 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:49.638620 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 10:38:49.638921 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":144,"threads_started":1}
I20260924 10:38:49.642120 19912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d94d49a1fd4ef8855df959c74b2575" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38669 } }
I20260924 10:38:49.642405 19912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:49.642472 19912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d94d49a1fd4ef8855df959c74b2575, State: Initialized, Role: FOLLOWER
I20260924 10:38:49.642923 19912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d94d49a1fd4ef8855df959c74b2575" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38669 } }
I20260924 10:38:49.643077 19912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:49.643141 19912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:49.643260 19912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:49.644198 19912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d94d49a1fd4ef8855df959c74b2575" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38669 } }
I20260924 10:38:49.644558 19912 leader_election.cc:302] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d94d49a1fd4ef8855df959c74b2575; no voters:
I20260924 10:38:49.644903 19912 leader_election.cc:288] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:38:49.645337 19917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:49.645594 19917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [term 1 LEADER]: Becoming Leader. State: Replica: 49d94d49a1fd4ef8855df959c74b2575, State: Running, Role: LEADER
I20260924 10:38:49.645907 19917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d94d49a1fd4ef8855df959c74b2575" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38669 } }
I20260924 10:38:49.646286 19912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:49.647526 19918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d94d49a1fd4ef8855df959c74b2575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d94d49a1fd4ef8855df959c74b2575" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38669 } } }
I20260924 10:38:49.647667 19918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:49.647886 19919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d94d49a1fd4ef8855df959c74b2575. Latest consensus state: current_term: 1 leader_uuid: "49d94d49a1fd4ef8855df959c74b2575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d94d49a1fd4ef8855df959c74b2575" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 38669 } } }
I20260924 10:38:49.648047 19919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:49.648099 19926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:49.650933 19926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:49.657024 19926 catalog_manager.cc:1347] Generated new cluster ID: 995c087309374a1c8919624cbee483ae
I20260924 10:38:49.657101 19926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:49.670150 19926 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:38:49.671418 19926 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:49.685334 19926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575: Generated new TSK 0
I20260924 10:38:49.686174 19926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:38:49.722640 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:38669
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:49.840631 19938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:49.840955 19938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:49.841042 19938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:49.845125 19938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:49.845263 19938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:49.845360 19938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:49.850901 19938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:49.852018 19938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:49.853091 19938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:49.862918 19944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:49.862944 19938 server_base.cc:1034] running on GCE node
W20260924 10:38:49.862902 19943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:49.862908 19946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:49.863579 19938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:49.864220 19938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:49.865404 19938 hybrid_clock.cc:648] HybridClock initialized: now 1790246329865372 us; error 49 us; skew 500 ppm
Sep 24 10:38:49 dist-test-slave-8wc7 krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246329, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:49.868163 19938 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:38:49.869029 19938 webserver.cc:460] Webserver started at http://127.2.193.129:33585/ using document root <none> and password file <none>
I20260924 10:38:49.869675 19938 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:49.869769 19938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:49.870003 19938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:49.872057 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "9eb168791a0f4afdae6d7b93c779e57e"
format_stamp: "Formatted at 2026-09-24 10:38:49 on dist-test-slave-8wc7"
server_key: "c82ddf0f6cefa8dcd80f22c68fb5eeff"
server_key_iv: "b1ab64cfb0444cfac66e112f4d059e87"
server_key_version: "encryptionkey@0"
I20260924 10:38:49.872637 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "9eb168791a0f4afdae6d7b93c779e57e"
format_stamp: "Formatted at 2026-09-24 10:38:49 on dist-test-slave-8wc7"
server_key: "c82ddf0f6cefa8dcd80f22c68fb5eeff"
server_key_iv: "b1ab64cfb0444cfac66e112f4d059e87"
server_key_version: "encryptionkey@0"
I20260924 10:38:49.876369 19938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:38:49.878895 19953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:49.880132 19938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:38:49.880275 19938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "9eb168791a0f4afdae6d7b93c779e57e"
format_stamp: "Formatted at 2026-09-24 10:38:49 on dist-test-slave-8wc7"
server_key: "c82ddf0f6cefa8dcd80f22c68fb5eeff"
server_key_iv: "b1ab64cfb0444cfac66e112f4d059e87"
server_key_version: "encryptionkey@0"
I20260924 10:38:49.880393 19938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:49.906535 19938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:49.910054 19938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:49.910265 19938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:49.910880 19938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:49.911723 19938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:49.911795 19938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:49.911866 19938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:49.911917 19938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:49.921002 19938 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:34679
I20260924 10:38:49.921025 20066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:34679 every 8 connection(s)
I20260924 10:38:49.921917 19938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:49.929713 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 19938
I20260924 10:38:49.929874 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:38:49.930233 2822 external_mini_cluster.cc:1428] Setting key e207f52546c582f6f22508eca59fc4d5
Sep 24 10:38:49 dist-test-slave-8wc7 krb5kdc[19831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246329, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:49.936355 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:49.923759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:58499 (local address 127.2.193.190:38669)
0924 10:38:49.923898 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:49.923902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:49.924767 (+ 865us) server_negotiation.cc:407] Connection header received
0924 10:38:49.925986 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:49.925989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:49.926052 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:49.926137 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:49.927335 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:49.927909 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:49.929103 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:49.929439 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:49.929558 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:49.932342 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:49.932348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:49.932354 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:49.932387 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:49.933971 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:49.934614 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:49.934616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:49.934618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:49.934694 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:49.935102 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:49.935104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:49.935105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:49.935279 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:49.935355 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:49.936054 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 10:38:49.936179 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:38:49.937629 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:49.924144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:38669 (local address 127.2.193.129:58499)
0924 10:38:49.924602 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:49.924647 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:49.925722 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:49.926313 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:49.926321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:49.926681 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:49.927171 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:49.927184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:49.928056 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:49.928060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:49.928958 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:49.928967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:49.929601 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:49.929606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:49.929778 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:49.930528 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:49.930545 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:49.932166 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:49.934091 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:49.934100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:49.934120 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:49.934495 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:49.934782 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:49.934788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:49.934793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:49.935007 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:49.935383 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:49.935393 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:49.935751 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 10:38:49.935988 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 10:38:49.936399 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260924 10:38:49.939038 20067 heartbeater.cc:347] Connected to a master server at 127.2.193.190:38669
I20260924 10:38:49.939311 20067 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:49.939828 20067 heartbeater.cc:511] Master 127.2.193.190:38669 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:38:49.941536 19876 ts_manager.cc:194] Registered new tserver with Master: 9eb168791a0f4afdae6d7b93c779e57e (127.2.193.129:34679)
I20260924 10:38:49.942721 19876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:58499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:38:49.959583 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:38669
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:50.074872 20072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:50.075181 20072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:50.075274 20072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:50.079311 20072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:50.079429 20072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:50.079524 20072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:50.085049 20072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:50.086256 20072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:50.087342 20072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:50.097362 20080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:50.097401 20072 server_base.cc:1034] running on GCE node
W20260924 10:38:50.097362 20078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:50.097373 20077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:50.098061 20072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:50.098758 20072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:50.099932 20072 hybrid_clock.cc:648] HybridClock initialized: now 1790246330099915 us; error 28 us; skew 500 ppm
Sep 24 10:38:50 dist-test-slave-8wc7 krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:50.102835 20072 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:38:50.103727 20072 webserver.cc:460] Webserver started at http://127.2.193.130:35887/ using document root <none> and password file <none>
I20260924 10:38:50.104269 20072 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:50.104332 20072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:50.104514 20072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:50.106727 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "ab1ab9294af14d1aafe910ddfe3f1233"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "79f5ec1f62fd1d020cc76d71d9d16893"
server_key_iv: "be8fa6451bab374eb59af0152ade9452"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.107262 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "ab1ab9294af14d1aafe910ddfe3f1233"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "79f5ec1f62fd1d020cc76d71d9d16893"
server_key_iv: "be8fa6451bab374eb59af0152ade9452"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.111104 20072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:38:50.113673 20087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:50.114972 20072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:50.115118 20072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "ab1ab9294af14d1aafe910ddfe3f1233"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "79f5ec1f62fd1d020cc76d71d9d16893"
server_key_iv: "be8fa6451bab374eb59af0152ade9452"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.115242 20072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:50.126410 20072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:50.130106 20072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:50.130275 20072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:50.130846 20072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:50.131641 20072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:50.131690 20072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:50.131729 20072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:50.131744 20072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:50.141322 20072 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:34381
I20260924 10:38:50.141341 20200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:34381 every 8 connection(s)
I20260924 10:38:50.142278 20072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:50.146114 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 20072
I20260924 10:38:50.146267 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:38:50.146607 2822 external_mini_cluster.cc:1428] Setting key 53dfc63548d7372826ed475bf3fb42b9
Sep 24 10:38:50 dist-test-slave-8wc7 krb5kdc[19831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:38:50.157920 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.144314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:41301 (local address 127.2.193.190:38669)
0924 10:38:50.144509 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.144514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.145404 (+ 890us) server_negotiation.cc:407] Connection header received
0924 10:38:50.146575 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.146578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.146630 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.146776 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:50.148318 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.149123 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.150628 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.150847 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.150945 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.153984 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:50.153988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:50.153991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:50.154018 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:50.155537 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:50.156055 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:50.156059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:50.156061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:50.156142 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:50.156538 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:50.156575 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:50.156577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:50.156896 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:50.157026 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.157686 (+ 660us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.157793 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
I20260924 10:38:50.158958 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.144739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:38669 (local address 127.2.193.130:41301)
0924 10:38:50.145255 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.145300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:50.146319 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:50.147024 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:50.147035 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:50.147556 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:50.148150 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.148174 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.149283 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.149286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.150495 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.150508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.151007 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.151012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.151263 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.151935 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:50.151961 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:50.153800 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:50.155666 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:50.155672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:50.155688 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:50.155950 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:50.156241 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:50.156244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:50.156246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:50.156433 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:50.157029 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:50.157038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:50.157436 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 10:38:50.157632 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 10:38:50.157899 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260924 10:38:50.160142 20201 heartbeater.cc:347] Connected to a master server at 127.2.193.190:38669
I20260924 10:38:50.160409 20201 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:50.160946 20201 heartbeater.cc:511] Master 127.2.193.190:38669 requested a full tablet report, sending...
I20260924 10:38:50.162046 19876 ts_manager.cc:194] Registered new tserver with Master: ab1ab9294af14d1aafe910ddfe3f1233 (127.2.193.130:34381)
I20260924 10:38:50.162618 19876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:41301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:38:50.175583 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:38669
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:50.290704 20206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:50.290973 20206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:50.291018 20206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:50.295429 20206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:50.295495 20206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:50.295560 20206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:50.300966 20206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:38669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:50.302052 20206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:50.303033 20206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:50.312841 20211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:50.312884 20206 server_base.cc:1034] running on GCE node
W20260924 10:38:50.312925 20214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:50.312925 20212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:50.313555 20206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:50.314198 20206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:50.315362 20206 hybrid_clock.cc:648] HybridClock initialized: now 1790246330315348 us; error 25 us; skew 500 ppm
Sep 24 10:38:50 dist-test-slave-8wc7 krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:50.318270 20206 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:38:50.319142 20206 webserver.cc:460] Webserver started at http://127.2.193.131:43873/ using document root <none> and password file <none>
I20260924 10:38:50.319793 20206 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:50.319885 20206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:50.320128 20206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:50.322237 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "78cf861c779a44bf8a98dd736be57235"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "8a97cf77c1a55548168e1921b452cf16"
server_key_iv: "aba3b5db2784954a1db0fa41bcb81bc5"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.322901 20206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "78cf861c779a44bf8a98dd736be57235"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "8a97cf77c1a55548168e1921b452cf16"
server_key_iv: "aba3b5db2784954a1db0fa41bcb81bc5"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.326613 20206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:38:50.329033 20221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:50.330262 20206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:38:50.330402 20206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "78cf861c779a44bf8a98dd736be57235"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "8a97cf77c1a55548168e1921b452cf16"
server_key_iv: "aba3b5db2784954a1db0fa41bcb81bc5"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.330523 20206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:50.351857 20206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:50.355156 20206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:50.355357 20206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:50.355926 20206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:50.356719 20206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:50.356786 20206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:50.356856 20206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:50.356907 20206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:50.365698 20206 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:41115
I20260924 10:38:50.365780 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:41115 every 8 connection(s)
I20260924 10:38:50.366621 20206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:50.371807 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 20206
I20260924 10:38:50.371948 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:38:50.372236 2822 external_mini_cluster.cc:1428] Setting key a0bde55deb8f7f623ca4330b9e78e53c
Sep 24 10:38:50 dist-test-slave-8wc7 krb5kdc[19831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246330, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:50.379765 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.368415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:34071 (local address 127.2.193.190:38669)
0924 10:38:50.368554 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.368558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.369468 (+ 910us) server_negotiation.cc:407] Connection header received
0924 10:38:50.370377 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.370380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.370437 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.370518 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:50.371685 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.372351 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.373796 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.374024 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.374144 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.376392 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:50.376395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:50.376397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:50.376421 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:50.377927 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:50.378401 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:50.378405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:50.378407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:50.378501 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:50.378808 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:50.378811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:50.378812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:50.378986 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:50.379061 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.379534 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.379637 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:38:50.380513 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.368860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:38669 (local address 127.2.193.131:34071)
0924 10:38:50.369319 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.369361 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:50.370170 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:50.370686 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:50.370693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:50.371048 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:50.371522 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.371536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.372480 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.372485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.373689 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.373699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.374188 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.374191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.374329 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.374828 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:50.374844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:50.376253 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:50.378025 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:50.378029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:50.378042 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:50.378288 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:50.378600 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:50.378602 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:50.378603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:50.378726 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:50.379073 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:50.379077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:50.379284 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 10:38:50.379445 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 10:38:50.379695 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260924 10:38:50.381634 20335 heartbeater.cc:347] Connected to a master server at 127.2.193.190:38669
I20260924 10:38:50.381872 20335 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:50.382344 20335 heartbeater.cc:511] Master 127.2.193.190:38669 requested a full tablet report, sending...
I20260924 10:38:50.383394 19876 ts_manager.cc:194] Registered new tserver with Master: 78cf861c779a44bf8a98dd736be57235 (127.2.193.131:41115)
I20260924 10:38:50.383955 19876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:34071
I20260924 10:38:50.385943 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:38:50.394833 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.387856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.2.193.190:38669)
0924 10:38:50.387993 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.387998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.388154 (+ 156us) server_negotiation.cc:407] Connection header received
0924 10:38:50.388253 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.388255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.388289 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.388345 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:50.388962 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.389504 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.390284 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.390482 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.390552 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.391752 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:50.391755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:50.391757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:50.391775 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:50.393287 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:50.393723 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:50.393725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:50.393727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:50.393794 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:50.394077 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:50.394079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:50.394080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:50.394298 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:50.394403 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.394597 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.394682 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:38:50.397660 19876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:38:50.400588 19876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:38:50.413832 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.408331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41115 (local address 127.0.0.1:60866)
0924 10:38:50.409159 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.409232 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:50.409365 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:50.410001 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:50.410007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:50.410030 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:50.410167 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.410178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.411299 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.411304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.412493 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.412502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.413325 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.413328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.413430 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.413469 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:38:50.413579 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:38:50.413685 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":30,"thread_start_us":57,"threads_started":1}
I20260924 10:38:50.413844 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.408471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.2.193.131:41115)
0924 10:38:50.408925 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.408937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.409509 (+ 572us) server_negotiation.cc:407] Connection header received
0924 10:38:50.409603 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.409609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.409830 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.409986 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:50.410301 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.411172 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.412610 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.413222 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.413322 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.413462 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.413615 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.413705 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":104,"threads_started":1}
I20260924 10:38:50.414565 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.408506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34381 (local address 127.0.0.1:57896)
0924 10:38:50.409355 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.409367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:50.409440 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:50.410441 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:50.410444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:50.410462 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:50.410566 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.410572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.411898 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.411901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.413116 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.413127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.414199 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.414202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.414288 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.414309 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:38:50.414368 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:38:50.414437 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":26,"thread_start_us":95,"threads_started":1}
I20260924 10:38:50.414611 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.408932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.2.193.130:34381)
0924 10:38:50.409942 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.409953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.409974 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:38:50.410043 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.410058 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.410259 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.410445 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:50.410685 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.411766 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.413257 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.414081 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.414199 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.414328 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.414388 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.414478 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":147,"threads_started":1}
I20260924 10:38:50.414799 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.408996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:34679 (local address 127.0.0.1:37594)
0924 10:38:50.409620 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.409631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:50.409701 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:50.410228 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:50.410231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:50.410248 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:50.410357 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.410367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.411828 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.411834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.413120 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.413127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.414401 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.414405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.414508 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.414532 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:38:50.414613 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:38:50.414678 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":45,"thread_start_us":82,"threads_started":1}
I20260924 10:38:50.415266 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.409428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.2.193.129:34679)
0924 10:38:50.409794 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.409807 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.409824 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:38:50.409885 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.409889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.410083 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.410239 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:50.410488 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.411677 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.413303 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.414118 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.414213 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.414341 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.415025 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.415124 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":101,"threads_started":1}
I20260924 10:38:50.416996 20269 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3eecd6387c4395acaba34a4c0ca28f (DEFAULT_TABLE table=test-table [id=0ceac90271444f7b915bbdbf748cc085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:50.416994 20135 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3eecd6387c4395acaba34a4c0ca28f (DEFAULT_TABLE table=test-table [id=0ceac90271444f7b915bbdbf748cc085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:50.418012 20135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3eecd6387c4395acaba34a4c0ca28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:50.418022 20269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3eecd6387c4395acaba34a4c0ca28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:50.424484 20353 tablet_bootstrap.cc:492] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Bootstrap starting.
I20260924 10:38:50.424950 20352 tablet_bootstrap.cc:492] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Bootstrap starting.
I20260924 10:38:50.425788 20001 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3eecd6387c4395acaba34a4c0ca28f (DEFAULT_TABLE table=test-table [id=0ceac90271444f7b915bbdbf748cc085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:50.426848 20001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3eecd6387c4395acaba34a4c0ca28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:50.427075 20353 tablet_bootstrap.cc:654] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:50.427277 20352 tablet_bootstrap.cc:654] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:50.427976 20352 log.cc:824] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:50.428236 20353 log.cc:824] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:50.429989 20353 tablet_bootstrap.cc:492] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: No bootstrap required, opened a new log
I20260924 10:38:50.429989 20352 tablet_bootstrap.cc:492] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: No bootstrap required, opened a new log
I20260924 10:38:50.430181 20352 ts_tablet_manager.cc:1397] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:38:50.430181 20353 ts_tablet_manager.cc:1397] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:38:50.433262 20356 tablet_bootstrap.cc:492] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e: Bootstrap starting.
I20260924 10:38:50.434101 20352 raft_consensus.cc:348] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.434101 20353 raft_consensus.cc:348] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.434386 20352 raft_consensus.cc:374] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:50.434386 20353 raft_consensus.cc:374] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:50.434444 20353 raft_consensus.cc:729] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cf861c779a44bf8a98dd736be57235, State: Initialized, Role: FOLLOWER
I20260924 10:38:50.434473 20352 raft_consensus.cc:729] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1ab9294af14d1aafe910ddfe3f1233, State: Initialized, Role: FOLLOWER
I20260924 10:38:50.434943 20353 consensus_queue.cc:260] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.434948 20352 consensus_queue.cc:260] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.435810 20335 heartbeater.cc:503] Master 127.2.193.190:38669 was elected leader, sending a full tablet report...
I20260924 10:38:50.435809 20201 heartbeater.cc:503] Master 127.2.193.190:38669 was elected leader, sending a full tablet report...
I20260924 10:38:50.435950 20352 ts_tablet_manager.cc:1428] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:38:50.435950 20353 ts_tablet_manager.cc:1428] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:38:50.437039 20356 tablet_bootstrap.cc:654] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:50.438558 20356 log.cc:824] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:50.440873 20356 tablet_bootstrap.cc:492] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e: No bootstrap required, opened a new log
I20260924 10:38:50.441085 20356 ts_tablet_manager.cc:1397] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 10:38:50.443909 20356 raft_consensus.cc:348] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.444134 20356 raft_consensus.cc:374] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:50.444208 20356 raft_consensus.cc:729] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb168791a0f4afdae6d7b93c779e57e, State: Initialized, Role: FOLLOWER
I20260924 10:38:50.444667 20356 consensus_queue.cc:260] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.445338 20067 heartbeater.cc:503] Master 127.2.193.190:38669 was elected leader, sending a full tablet report...
I20260924 10:38:50.445674 20356 ts_tablet_manager.cc:1428] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:38:50.576201 20360 raft_consensus.cc:482] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:50.576457 20360 raft_consensus.cc:504] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.577701 20360 leader_election.cc:288] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78cf861c779a44bf8a98dd736be57235 (127.2.193.131:41115), ab1ab9294af14d1aafe910ddfe3f1233 (127.2.193.130:34381)
I20260924 10:38:50.583505 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.578123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:42535 (local address 127.2.193.131:41115)
0924 10:38:50.578265 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.578272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.578653 (+ 381us) server_negotiation.cc:407] Connection header received
0924 10:38:50.578743 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.578747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.578799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.578887 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:50.579325 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.580114 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.582140 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.582765 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.582896 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.582948 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.583282 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.583373 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:38:50.583505 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.578211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:43473 (local address 127.2.193.130:34381)
0924 10:38:50.578373 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.578377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.578567 (+ 190us) server_negotiation.cc:407] Connection header received
0924 10:38:50.578744 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.578747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.578802 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.578888 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:50.579324 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.580114 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.582140 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.582765 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.582896 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.582948 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.583282 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.583373 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:38:50.583555 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.577985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34381 (local address 127.2.193.129:43473)
0924 10:38:50.578465 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.578482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:50.578613 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:50.578913 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:50.578916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:50.578957 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:50.579204 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.579211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.580237 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.580240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.582005 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.582019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.582911 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.582915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.583109 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.583171 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 10:38:50.583254 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:38:50.583361 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":263,"spinlock_wait_cycles":4736,"thread_start_us":133,"threads_started":1}
I20260924 10:38:50.583562 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.577985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:41115 (local address 127.2.193.129:42535)
0924 10:38:50.578563 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.578577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:50.578663 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:50.578913 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:50.578916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:50.578956 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:50.579204 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.579211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.580237 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.580240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.582005 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:50.582019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.582911 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.582915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:50.583125 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.583179 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 10:38:50.583254 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:38:50.583361 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":499,"spinlock_wait_cycles":39424,"thread_start_us":94,"threads_started":1}
I20260924 10:38:50.584322 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eecd6387c4395acaba34a4c0ca28f" candidate_uuid: "9eb168791a0f4afdae6d7b93c779e57e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" is_pre_election: true
I20260924 10:38:50.584334 20289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eecd6387c4395acaba34a4c0ca28f" candidate_uuid: "9eb168791a0f4afdae6d7b93c779e57e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cf861c779a44bf8a98dd736be57235" is_pre_election: true
I20260924 10:38:50.584614 20155 raft_consensus.cc:2454] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb168791a0f4afdae6d7b93c779e57e in term 0.
I20260924 10:38:50.584625 20289 raft_consensus.cc:2454] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb168791a0f4afdae6d7b93c779e57e in term 0.
I20260924 10:38:50.585150 19955 leader_election.cc:302] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 78cf861c779a44bf8a98dd736be57235, 9eb168791a0f4afdae6d7b93c779e57e, ab1ab9294af14d1aafe910ddfe3f1233; no voters:
I20260924 10:38:50.585467 20360 raft_consensus.cc:2781] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:38:50.585543 20360 raft_consensus.cc:482] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:38:50.585621 20360 raft_consensus.cc:3037] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:50.586702 20360 raft_consensus.cc:504] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.587097 20360 leader_election.cc:288] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [CANDIDATE]: Term 1 election: Requested vote from peers 78cf861c779a44bf8a98dd736be57235 (127.2.193.131:41115), ab1ab9294af14d1aafe910ddfe3f1233 (127.2.193.130:34381)
I20260924 10:38:50.587472 20289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eecd6387c4395acaba34a4c0ca28f" candidate_uuid: "9eb168791a0f4afdae6d7b93c779e57e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cf861c779a44bf8a98dd736be57235"
I20260924 10:38:50.587533 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3eecd6387c4395acaba34a4c0ca28f" candidate_uuid: "9eb168791a0f4afdae6d7b93c779e57e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1ab9294af14d1aafe910ddfe3f1233"
I20260924 10:38:50.587637 20289 raft_consensus.cc:3037] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:50.587679 20155 raft_consensus.cc:3037] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:50.588708 20155 raft_consensus.cc:2454] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb168791a0f4afdae6d7b93c779e57e in term 1.
I20260924 10:38:50.588716 20289 raft_consensus.cc:2454] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb168791a0f4afdae6d7b93c779e57e in term 1.
I20260924 10:38:50.589072 19957 leader_election.cc:302] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78cf861c779a44bf8a98dd736be57235, 9eb168791a0f4afdae6d7b93c779e57e; no voters:
I20260924 10:38:50.589336 20360 raft_consensus.cc:2781] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:50.589577 20360 raft_consensus.cc:686] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [term 1 LEADER]: Becoming Leader. State: Replica: 9eb168791a0f4afdae6d7b93c779e57e, State: Running, Role: LEADER
I20260924 10:38:50.589936 20360 consensus_queue.cc:237] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } }
I20260924 10:38:50.593082 19872 catalog_manager.cc:5495] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eb168791a0f4afdae6d7b93c779e57e (127.2.193.129). New cstate: current_term: 1 leader_uuid: "9eb168791a0f4afdae6d7b93c779e57e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eb168791a0f4afdae6d7b93c779e57e" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 34679 } health_report { overall_health: HEALTHY } } }
W20260924 10:38:50.617647 20336 tablet.cc:2369] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:38:50.643558 20202 tablet.cc:2369] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:50.649678 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.645867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.2.193.129:34679)
0924 10:38:50.646028 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.646033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.646060 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:38:50.646188 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.646190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.646247 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.646363 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:50.646610 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.647178 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.648178 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.648394 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.648503 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.648828 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:50.649289 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:38:50.649430 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.649471 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.649527 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:38:50.655707 20289 raft_consensus.cc:1261] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 1 FOLLOWER]: Refusing update from remote peer 9eb168791a0f4afdae6d7b93c779e57e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:50.655707 20155 raft_consensus.cc:1261] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 1 FOLLOWER]: Refusing update from remote peer 9eb168791a0f4afdae6d7b93c779e57e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:50.656459 20360 consensus_queue.cc:1035] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "78cf861c779a44bf8a98dd736be57235" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:50.656669 20363 consensus_queue.cc:1035] T 3c3eecd6387c4395acaba34a4c0ca28f P 9eb168791a0f4afdae6d7b93c779e57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1ab9294af14d1aafe910ddfe3f1233" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:50.662922 20366 mvcc.cc:205] Tried to move back new op lower bound from 7332848970357977088 to 7332848970099232768. Current Snapshot: MvccSnapshot[applied={T|T < 7332848970357977088}]
I20260924 10:38:50.666631 20368 mvcc.cc:205] Tried to move back new op lower bound from 7332848970357977088 to 7332848970099232768. Current Snapshot: MvccSnapshot[applied={T|T < 7332848970357977088 or (T in {7332848970357977088})}]
I20260924 10:38:50.672874 19872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:50.673082 19872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:50.675486 19872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49d94d49a1fd4ef8855df959c74b2575: Sending DeleteTablet for 3 replicas of tablet 3c3eecd6387c4395acaba34a4c0ca28f
I20260924 10:38:50.676303 20135 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3eecd6387c4395acaba34a4c0ca28f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:50 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:57896
I20260924 10:38:50.676301 20269 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3eecd6387c4395acaba34a4c0ca28f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:50 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:60866
I20260924 10:38:50.676357 20001 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3eecd6387c4395acaba34a4c0ca28f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:50 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:37594
I20260924 10:38:50.676780 20377 tablet_replica.cc:330] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: stopping tablet replica
I20260924 10:38:50.677129 20377 raft_consensus.cc:2229] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:50.677184 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 19938
I20260924 10:38:50.677500 20377 raft_consensus.cc:2258] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:50.679235 20377 ts_tablet_manager.cc:1905] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:50.679492 20378 tablet_replica.cc:330] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: stopping tablet replica
I20260924 10:38:50.679783 20378 raft_consensus.cc:2229] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:50.680060 20378 raft_consensus.cc:2258] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:50.681495 20378 ts_tablet_manager.cc:1905] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:50.682034 20377 ts_tablet_manager.cc:1918] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:50.682128 20377 log.cc:1196] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/3c3eecd6387c4395acaba34a4c0ca28f
I20260924 10:38:50.682466 20377 ts_tablet_manager.cc:1939] T 3c3eecd6387c4395acaba34a4c0ca28f P 78cf861c779a44bf8a98dd736be57235: Deleting consensus metadata
I20260924 10:38:50.683735 19863 catalog_manager.cc:4841] TS 78cf861c779a44bf8a98dd736be57235 (127.2.193.131:41115): tablet 3c3eecd6387c4395acaba34a4c0ca28f (table test-table [id=0ceac90271444f7b915bbdbf748cc085]) successfully deleted
I20260924 10:38:50.684149 20378 ts_tablet_manager.cc:1918] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:50.684266 20378 log.cc:1196] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/3c3eecd6387c4395acaba34a4c0ca28f
I20260924 10:38:50.684566 20378 ts_tablet_manager.cc:1939] T 3c3eecd6387c4395acaba34a4c0ca28f P ab1ab9294af14d1aafe910ddfe3f1233: Deleting consensus metadata
W20260924 10:38:50.684526 19861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:34679 (error 108)
I20260924 10:38:50.685443 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 20072
I20260924 10:38:50.685657 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.685316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:34679 (local address 127.0.0.1:37616)
0924 10:38:50.685465 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.685556 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 10:38:50.686998 19861 catalog_manager.cc:4584] TS 9eb168791a0f4afdae6d7b93c779e57e (127.2.193.129:34679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c3eecd6387c4395acaba34a4c0ca28f: Network error: Client connection negotiation failed: client connection to 127.2.193.129:34679: connect: Connection refused (error 111)
I20260924 10:38:50.691046 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 20206
I20260924 10:38:50.692718 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.691207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34381 (local address 127.0.0.1:57908)
0924 10:38:50.692560 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:50.692611 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:34381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1273}
I20260924 10:38:50.696623 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 19844
2026-09-24T10:38:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2442 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T10:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:38:50Z Disabled control of system clock
I20260924 10:38:50.726464 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:39235
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:39235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:38:50.837441 20383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:50.837730 20383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:50.837765 20383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:50.841728 20383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:50.841782 20383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:50.841800 20383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:50.841817 20383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:50.841832 20383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:50.846506 20383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:39235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:39235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:50.847563 20383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:50.848904 20383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:38:50.857496 20383 server_base.cc:1034] running on GCE node
W20260924 10:38:50.857442 20389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:50.857442 20388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:50.857462 20391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:50.858429 20383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:50.859534 20383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:50.860728 20383 hybrid_clock.cc:648] HybridClock initialized: now 1790246330860714 us; error 30 us; skew 500 ppm
I20260924 10:38:50.862634 20383 webserver.cc:460] Webserver started at http://127.2.193.190:39345/ using document root <none> and password file <none>
I20260924 10:38:50.863229 20383 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:50.863324 20383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:50.863543 20383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:50.865662 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "957cab28948342708c656b918a1769b6"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "e7231b7db3509b1734cc23657d689112"
server_key_iv: "5e2295a59cd46716c655b15cd2b508b4"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.866225 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "957cab28948342708c656b918a1769b6"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "e7231b7db3509b1734cc23657d689112"
server_key_iv: "5e2295a59cd46716c655b15cd2b508b4"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.869829 20383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:50.872110 20397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:50.873320 20383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:38:50.873461 20383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "957cab28948342708c656b918a1769b6"
format_stamp: "Formatted at 2026-09-24 10:38:50 on dist-test-slave-8wc7"
server_key: "e7231b7db3509b1734cc23657d689112"
server_key_iv: "5e2295a59cd46716c655b15cd2b508b4"
server_key_version: "encryptionkey@0"
I20260924 10:38:50.873574 20383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:50.884371 20383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:50.885015 20383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:50.885241 20383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:50.892002 20383 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:39235
I20260924 10:38:50.892017 20449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:39235 every 8 connection(s)
I20260924 10:38:50.892932 20383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:50.896004 20450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:50.901427 20450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6: Bootstrap starting.
I20260924 10:38:50.901865 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 20383
I20260924 10:38:50.901996 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:38:50.902267 2822 external_mini_cluster.cc:1428] Setting key cd093157997ab13d1ee6094f5742bb38
I20260924 10:38:50.904047 20450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:50.904778 20450 log.cc:824] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:50.906844 20450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6: No bootstrap required, opened a new log
I20260924 10:38:50.911140 20450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957cab28948342708c656b918a1769b6" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39235 } }
I20260924 10:38:50.911372 20450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:50.911458 20450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957cab28948342708c656b918a1769b6, State: Initialized, Role: FOLLOWER
I20260924 10:38:50.911633 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:50.903542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.2.193.190:39235)
0924 10:38:50.904212 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:50.904226 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:50.904261 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:38:50.904872 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:50.904915 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:50.905473 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:50.905799 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:50.906135 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.907158 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.908365 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:50.908818 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:50.908936 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:50.909797 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:50.909802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:50.909814 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:50.910203 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:50.910254 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:50.910272 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:50.910347 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:50.910522 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 10:38:50.910816 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260924 10:38:50.911975 20450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957cab28948342708c656b918a1769b6" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39235 } }
I20260924 10:38:50.912130 20450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:50.912192 20450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:50.912283 20450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:50.913292 20450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957cab28948342708c656b918a1769b6" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39235 } }
I20260924 10:38:50.913652 20450 leader_election.cc:302] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 957cab28948342708c656b918a1769b6; no voters:
I20260924 10:38:50.914008 20450 leader_election.cc:288] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:38:50.914125 20455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:50.914378 20455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [term 1 LEADER]: Becoming Leader. State: Replica: 957cab28948342708c656b918a1769b6, State: Running, Role: LEADER
I20260924 10:38:50.914697 20455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957cab28948342708c656b918a1769b6" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39235 } }
I20260924 10:38:50.915485 20450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:50.916365 20457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957cab28948342708c656b918a1769b6. Latest consensus state: current_term: 1 leader_uuid: "957cab28948342708c656b918a1769b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957cab28948342708c656b918a1769b6" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39235 } } }
I20260924 10:38:50.916507 20457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:50.916805 20456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "957cab28948342708c656b918a1769b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957cab28948342708c656b918a1769b6" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 39235 } } }
I20260924 10:38:50.916953 20456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:50.917827 20463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:50.920521 20463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:50.925685 20463 catalog_manager.cc:1347] Generated new cluster ID: 7fb351dc728a4ea795daff1bb0a1145d
I20260924 10:38:50.925750 20463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:50.937645 20463 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:38:50.938891 20463 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:50.947028 20463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6: Generated new TSK 0
I20260924 10:38:50.947882 20463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:38:50.952950 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39235
--builtin_ntp_servers=127.2.193.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:38:51.067873 20474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:51.068199 20474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:51.068280 20474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:51.072335 20474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:51.072394 20474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:51.072453 20474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:51.077376 20474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:51.078644 20474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:51.080022 20474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:51.089795 20479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:51.089783 20480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:51.089932 20474 server_base.cc:1034] running on GCE node
W20260924 10:38:51.089783 20482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:51.090514 20474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:51.091284 20474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:51.092474 20474 hybrid_clock.cc:648] HybridClock initialized: now 1790246331092452 us; error 38 us; skew 500 ppm
I20260924 10:38:51.094552 20474 webserver.cc:460] Webserver started at http://127.2.193.129:44939/ using document root <none> and password file <none>
I20260924 10:38:51.095175 20474 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:51.095274 20474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:51.095538 20474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:51.097756 20474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "bd621870d1c044c1830d4b47886b2650"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "86ead735152e28dcb123844557a919c8"
server_key_iv: "d2c14940407fce4c4908affa62bdc774"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.098347 20474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "bd621870d1c044c1830d4b47886b2650"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "86ead735152e28dcb123844557a919c8"
server_key_iv: "d2c14940407fce4c4908affa62bdc774"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.102289 20474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 10:38:51.104828 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.106109 20474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:38:51.106251 20474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "bd621870d1c044c1830d4b47886b2650"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "86ead735152e28dcb123844557a919c8"
server_key_iv: "d2c14940407fce4c4908affa62bdc774"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.106375 20474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:51.119951 20474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:51.120656 20474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:51.120851 20474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:51.121469 20474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:51.122399 20474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:51.122468 20474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.122539 20474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:51.122584 20474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.131507 20474 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:35411
I20260924 10:38:51.131526 20601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:35411 every 8 connection(s)
I20260924 10:38:51.132434 20474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:51.139523 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 20474
I20260924 10:38:51.139650 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:38:51.139921 2822 external_mini_cluster.cc:1428] Setting key acc0fd1f3f0402f69b09ae6f7d8333e2
I20260924 10:38:51.142141 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39235
--builtin_ntp_servers=127.2.193.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:38:51.142513 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.134330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:54809 (local address 127.2.193.190:39235)
0924 10:38:51.134515 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.134520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.135625 (+ 1105us) server_negotiation.cc:407] Connection header received
0924 10:38:51.136984 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.136987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.137046 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.137178 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:51.138106 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.138654 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.140127 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.140419 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.140631 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.141920 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:51.141926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:51.141933 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:51.141949 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:51.141975 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:51.141986 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:51.142073 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.142226 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.142295 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:38:51.143469 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.134824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39235 (local address 127.2.193.129:54809)
0924 10:38:51.135474 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.135513 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.136759 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.137376 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.137383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.137469 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:51.137948 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.137960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.138781 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.138787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.139974 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.139986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.140703 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.140711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.140905 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.141629 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:51.141631 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:51.141675 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:51.141753 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:51.142080 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:51.142087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:51.142091 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.142189 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.142526 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260924 10:38:51.144790 20602 heartbeater.cc:347] Connected to a master server at 127.2.193.190:39235
I20260924 10:38:51.145036 20602 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:51.145714 20602 heartbeater.cc:511] Master 127.2.193.190:39235 requested a full tablet report, sending...
I20260924 10:38:51.147332 20414 ts_manager.cc:194] Registered new tserver with Master: bd621870d1c044c1830d4b47886b2650 (127.2.193.129:35411)
I20260924 10:38:51.148551 20414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.129:54809
W20260924 10:38:51.254402 20605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:51.254664 20605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:51.254703 20605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:51.258773 20605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:51.258828 20605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:51.258930 20605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:51.263752 20605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:51.264868 20605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:51.266286 20605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:51.276099 20613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:51.276110 20611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:51.276144 20605 server_base.cc:1034] running on GCE node
W20260924 10:38:51.276099 20610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:51.276682 20605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:51.277438 20605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:51.278595 20605 hybrid_clock.cc:648] HybridClock initialized: now 1790246331278543 us; error 64 us; skew 500 ppm
I20260924 10:38:51.280431 20605 webserver.cc:460] Webserver started at http://127.2.193.130:44099/ using document root <none> and password file <none>
I20260924 10:38:51.281002 20605 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:51.281064 20605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:51.281327 20605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:51.283360 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "a74f8efca8394f26b71e5f516a9b2f82"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "2de8f7e75bd3bfd271722933c3ae301e"
server_key_iv: "055ef5a26d5fc94c47f4e0ca6a6ad1b7"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.283924 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "a74f8efca8394f26b71e5f516a9b2f82"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "2de8f7e75bd3bfd271722933c3ae301e"
server_key_iv: "055ef5a26d5fc94c47f4e0ca6a6ad1b7"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.287600 20605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:38:51.290060 20619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.291266 20605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:38:51.291386 20605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "a74f8efca8394f26b71e5f516a9b2f82"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "2de8f7e75bd3bfd271722933c3ae301e"
server_key_iv: "055ef5a26d5fc94c47f4e0ca6a6ad1b7"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.291493 20605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:51.311214 20605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:51.311908 20605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:51.312110 20605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:51.312711 20605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:51.313714 20605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:51.313758 20605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.313828 20605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:51.313870 20605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.322960 20605 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:32811
I20260924 10:38:51.322981 20732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:32811 every 8 connection(s)
I20260924 10:38:51.323870 20605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:51.329260 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 20605
I20260924 10:38:51.329355 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:38:51.329660 2822 external_mini_cluster.cc:1428] Setting key 07c2ddcd71f995f85b580319e9841a34
I20260924 10:38:51.331610 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39235
--builtin_ntp_servers=127.2.193.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:38:51.333560 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.325858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:59917 (local address 127.2.193.190:39235)
0924 10:38:51.326036 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.326042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.327020 (+ 978us) server_negotiation.cc:407] Connection header received
0924 10:38:51.328464 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.328468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.328534 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.328638 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:51.329719 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.330281 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.331459 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.331746 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.331849 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.333044 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:51.333048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:51.333051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:51.333071 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:51.333089 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:51.333096 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:51.333171 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.333361 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.333422 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 10:38:51.334414 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.326220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39235 (local address 127.2.193.130:59917)
0924 10:38:51.326856 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.326898 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.328242 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.328817 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.328823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.328910 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:51.329569 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.329585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.330409 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.330413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.331327 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.331336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.331985 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.331988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.332134 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.332779 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:51.332781 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:51.332825 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:51.332890 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:51.333206 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:51.333219 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:51.333228 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.333350 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.333658 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260924 10:38:51.335592 20733 heartbeater.cc:347] Connected to a master server at 127.2.193.190:39235
I20260924 10:38:51.335860 20733 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:51.336531 20733 heartbeater.cc:511] Master 127.2.193.190:39235 requested a full tablet report, sending...
I20260924 10:38:51.337755 20414 ts_manager.cc:194] Registered new tserver with Master: a74f8efca8394f26b71e5f516a9b2f82 (127.2.193.130:32811)
I20260924 10:38:51.338645 20414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.130:59917
W20260924 10:38:51.445400 20736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:51.445686 20736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:51.445726 20736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:51.450173 20736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:51.450243 20736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:51.450302 20736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:51.455370 20736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:51.456466 20736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:51.457922 20736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:51.467814 20744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:51.467772 20742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:51.467921 20736 server_base.cc:1034] running on GCE node
W20260924 10:38:51.467775 20741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:51.468614 20736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:51.469286 20736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:51.470446 20736 hybrid_clock.cc:648] HybridClock initialized: now 1790246331470411 us; error 51 us; skew 500 ppm
I20260924 10:38:51.472326 20736 webserver.cc:460] Webserver started at http://127.2.193.131:39541/ using document root <none> and password file <none>
I20260924 10:38:51.472922 20736 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:51.472986 20736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:51.473269 20736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:51.475433 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "f00c5337042647c8ba05b2a65a19ca8b"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "5ea589b12b1800fcbed68bb241bfcd26"
server_key_iv: "770f04b9869810d60f55702084792eba"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.476020 20736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "f00c5337042647c8ba05b2a65a19ca8b"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "5ea589b12b1800fcbed68bb241bfcd26"
server_key_iv: "770f04b9869810d60f55702084792eba"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.479964 20736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:38:51.482475 20750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.483740 20736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:38:51.483842 20736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "f00c5337042647c8ba05b2a65a19ca8b"
format_stamp: "Formatted at 2026-09-24 10:38:51 on dist-test-slave-8wc7"
server_key: "5ea589b12b1800fcbed68bb241bfcd26"
server_key_iv: "770f04b9869810d60f55702084792eba"
server_key_version: "encryptionkey@0"
I20260924 10:38:51.483991 20736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:51.493793 20736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:51.494555 20736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:51.494742 20736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:51.495400 20736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:51.496373 20736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:51.496425 20736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.496493 20736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:51.496527 20736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:51.505229 20736 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:34733
I20260924 10:38:51.505259 20863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:34733 every 8 connection(s)
I20260924 10:38:51.506134 20736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:51.508358 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 20736
I20260924 10:38:51.508473 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:38:51.508821 2822 external_mini_cluster.cc:1428] Setting key 748fa39b01322ad694fca1986b95e70c
I20260924 10:38:51.515875 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.507946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:56817 (local address 127.2.193.190:39235)
0924 10:38:51.508094 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.508097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.509384 (+ 1287us) server_negotiation.cc:407] Connection header received
0924 10:38:51.511085 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.511088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.511146 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.511242 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:51.512136 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.512679 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.513917 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.514150 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.514258 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.515318 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:51.515322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:51.515325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:51.515337 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:51.515363 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:51.515371 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:51.515445 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.515643 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.515707 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:38:51.516673 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.508359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:39235 (local address 127.2.193.131:56817)
0924 10:38:51.509212 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.509267 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.510842 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.511423 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.511431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.511517 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:51.512003 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.512014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.512805 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.512808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.513782 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.513792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.514293 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.514297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.514471 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.515066 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:51.515067 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:51.515107 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:51.515172 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:51.515460 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:51.515468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:51.515474 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.515609 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.515894 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":149,"threads_started":1}
I20260924 10:38:51.517889 20864 heartbeater.cc:347] Connected to a master server at 127.2.193.190:39235
I20260924 10:38:51.518180 20864 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:51.518704 20864 heartbeater.cc:511] Master 127.2.193.190:39235 requested a full tablet report, sending...
I20260924 10:38:51.519747 20414 ts_manager.cc:194] Registered new tserver with Master: f00c5337042647c8ba05b2a65a19ca8b (127.2.193.131:34733)
I20260924 10:38:51.520292 20414 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.131:56817
I20260924 10:38:51.522715 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:38:51.528129 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.524795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.2.193.190:39235)
0924 10:38:51.524910 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.524920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.524933 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:38:51.525007 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.525009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.525045 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.525113 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:51.525416 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.525916 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.526758 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.526957 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.527031 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.527782 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:51.527785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:51.527787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:51.527797 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:51.527816 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:51.527823 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:51.527879 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.527972 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.528020 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:38:51.530982 20414 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:38:51.533979 20414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:38:51.548915 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.542857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35411 (local address 127.0.0.1:43372)
0924 10:38:51.543869 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.543882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.543973 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.544648 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.544650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.544662 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:51.544760 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.544765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.545857 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.545860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.547539 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.547550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.548355 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.548359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.548464 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.548499 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.548618 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.548746 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":175,"thread_start_us":132,"threads_started":1}
I20260924 10:38:51.548919 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.543213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.2.193.129:35411)
0924 10:38:51.543595 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.543607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.544084 (+ 477us) server_negotiation.cc:407] Connection header received
0924 10:38:51.544169 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.544174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.544344 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.544481 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:51.544902 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.545720 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.547683 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.548229 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.548355 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.548565 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.548656 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.548757 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260924 10:38:51.549325 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.542485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.2.193.130:32811)
0924 10:38:51.542942 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.542958 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.543491 (+ 533us) server_negotiation.cc:407] Connection header received
0924 10:38:51.543598 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.543607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.543841 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.544006 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:51.544476 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.545699 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.547667 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.548438 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.548533 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.548677 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.549036 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.549135 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":144,"threads_started":1}
I20260924 10:38:51.549108 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.543000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:34733 (local address 127.0.0.1:58546)
0924 10:38:51.544002 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.544015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.544109 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.544595 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.544597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.544616 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:51.544753 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.544759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.546077 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.546097 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.547281 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.547291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.548670 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.548673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.548767 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.548797 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.548869 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.548958 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":77,"thread_start_us":97,"threads_started":1}
I20260924 10:38:51.549700 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.542277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:32811 (local address 127.0.0.1:60034)
0924 10:38:51.543155 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.543210 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.543345 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.544184 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.544190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.544211 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:51.544338 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.544344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.545831 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.545835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.547539 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.547556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.548725 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.548728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.548814 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.548848 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.548907 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.548958 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":220,"thread_start_us":91,"threads_started":1}
I20260924 10:38:51.549955 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.543425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58546 (local address 127.2.193.131:34733)
0924 10:38:51.543803 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.543818 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.544196 (+ 378us) server_negotiation.cc:407] Connection header received
0924 10:38:51.544261 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.544264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.544437 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.544614 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:51.544873 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.545972 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.547502 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.548303 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.548413 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.548605 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.549227 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.549780 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":171,"threads_started":1}
I20260924 10:38:51.552476 20667 tablet_service.cc:1467] Processing CreateTablet for tablet 721f1e7c4c3c4a8da1feaf7d6b123495 (DEFAULT_TABLE table=test-table [id=54c044e5032947859d152759b6c97a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:51.552518 20536 tablet_service.cc:1467] Processing CreateTablet for tablet 721f1e7c4c3c4a8da1feaf7d6b123495 (DEFAULT_TABLE table=test-table [id=54c044e5032947859d152759b6c97a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:51.552577 20798 tablet_service.cc:1467] Processing CreateTablet for tablet 721f1e7c4c3c4a8da1feaf7d6b123495 (DEFAULT_TABLE table=test-table [id=54c044e5032947859d152759b6c97a5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:51.553599 20798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721f1e7c4c3c4a8da1feaf7d6b123495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:51.553599 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721f1e7c4c3c4a8da1feaf7d6b123495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:51.553663 20667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721f1e7c4c3c4a8da1feaf7d6b123495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:51.560487 20881 tablet_bootstrap.cc:492] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Bootstrap starting.
I20260924 10:38:51.561832 20883 tablet_bootstrap.cc:492] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650: Bootstrap starting.
I20260924 10:38:51.562850 20881 tablet_bootstrap.cc:654] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:51.563580 20881 log.cc:824] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:51.563608 20883 tablet_bootstrap.cc:654] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:51.564111 20882 tablet_bootstrap.cc:492] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Bootstrap starting.
I20260924 10:38:51.564459 20883 log.cc:824] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:51.565794 20881 tablet_bootstrap.cc:492] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: No bootstrap required, opened a new log
I20260924 10:38:51.566001 20881 ts_tablet_manager.cc:1397] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:38:51.566231 20882 tablet_bootstrap.cc:654] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:51.566659 20883 tablet_bootstrap.cc:492] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650: No bootstrap required, opened a new log
I20260924 10:38:51.566846 20883 ts_tablet_manager.cc:1397] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:38:51.567054 20882 log.cc:824] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:51.568990 20882 tablet_bootstrap.cc:492] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: No bootstrap required, opened a new log
I20260924 10:38:51.569192 20882 ts_tablet_manager.cc:1397] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:38:51.570144 20883 raft_consensus.cc:348] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.570386 20883 raft_consensus.cc:374] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:51.570441 20883 raft_consensus.cc:729] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd621870d1c044c1830d4b47886b2650, State: Initialized, Role: FOLLOWER
I20260924 10:38:51.570425 20881 raft_consensus.cc:348] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.570675 20881 raft_consensus.cc:374] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:51.570724 20881 raft_consensus.cc:729] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74f8efca8394f26b71e5f516a9b2f82, State: Initialized, Role: FOLLOWER
I20260924 10:38:51.570906 20883 consensus_queue.cc:260] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.571183 20881 consensus_queue.cc:260] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.571949 20883 ts_tablet_manager.cc:1428] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:38:51.572137 20881 ts_tablet_manager.cc:1428] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:38:51.572326 20602 heartbeater.cc:503] Master 127.2.193.190:39235 was elected leader, sending a full tablet report...
I20260924 10:38:51.572487 20733 heartbeater.cc:503] Master 127.2.193.190:39235 was elected leader, sending a full tablet report...
I20260924 10:38:51.573498 20882 raft_consensus.cc:348] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.573757 20882 raft_consensus.cc:374] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:51.573817 20882 raft_consensus.cc:729] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00c5337042647c8ba05b2a65a19ca8b, State: Initialized, Role: FOLLOWER
I20260924 10:38:51.574221 20882 consensus_queue.cc:260] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
W20260924 10:38:51.574790 20734 tablet.cc:2369] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:51.574880 20864 heartbeater.cc:503] Master 127.2.193.190:39235 was elected leader, sending a full tablet report...
I20260924 10:38:51.575145 20882 ts_tablet_manager.cc:1428] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 10:38:51.633561 20603 tablet.cc:2369] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:51.734807 20888 raft_consensus.cc:482] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:51.735070 20888 raft_consensus.cc:504] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.736730 20888 leader_election.cc:288] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00c5337042647c8ba05b2a65a19ca8b (127.2.193.131:34733), bd621870d1c044c1830d4b47886b2650 (127.2.193.129:35411)
I20260924 10:38:51.740765 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.736956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:34733 (local address 127.2.193.130:43535)
0924 10:38:51.737107 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.737125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.737250 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.737770 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.737772 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.737786 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:51.737943 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.737965 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.738874 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.738880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.739684 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.739693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.740329 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.740332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.740430 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.740457 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.740547 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.740648 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":1}
I20260924 10:38:51.740765 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.737295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:43535 (local address 127.2.193.131:34733)
0924 10:38:51.737413 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.737417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.737444 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:38:51.737511 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.737517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.737571 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.737656 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:51.738100 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.738753 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.739811 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.740208 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.740342 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.740392 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.740561 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.740632 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:38:51.741772 20818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721f1e7c4c3c4a8da1feaf7d6b123495" candidate_uuid: "a74f8efca8394f26b71e5f516a9b2f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00c5337042647c8ba05b2a65a19ca8b" is_pre_election: true
I20260924 10:38:51.742100 20818 raft_consensus.cc:2454] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74f8efca8394f26b71e5f516a9b2f82 in term 0.
I20260924 10:38:51.742625 20620 leader_election.cc:302] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74f8efca8394f26b71e5f516a9b2f82, f00c5337042647c8ba05b2a65a19ca8b; no voters:
I20260924 10:38:51.742844 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.737210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:44339 (local address 127.2.193.129:35411)
0924 10:38:51.737362 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.737367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.737885 (+ 518us) server_negotiation.cc:407] Connection header received
0924 10:38:51.738019 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.738023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.738081 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.738150 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:51.738498 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.739369 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.741217 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.741962 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.742071 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.742132 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.742575 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.742685 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:38:51.742946 20888 raft_consensus.cc:2781] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:38:51.743026 20888 raft_consensus.cc:482] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:38:51.743057 20888 raft_consensus.cc:3037] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:51.743287 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.736952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35411 (local address 127.2.193.130:44339)
0924 10:38:51.737774 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.737791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:51.737908 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:51.738250 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:51.738253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:51.738268 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:51.738378 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.738383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.739523 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.739529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.741030 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:51.741044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.742271 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.742276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:51.742400 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.742432 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:38:51.742999 (+ 567us) client_negotiation.cc:240] Negotiation successful
0924 10:38:51.743091 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":465,"thread_start_us":116,"threads_started":1}
I20260924 10:38:51.743935 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721f1e7c4c3c4a8da1feaf7d6b123495" candidate_uuid: "a74f8efca8394f26b71e5f516a9b2f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd621870d1c044c1830d4b47886b2650" is_pre_election: true
I20260924 10:38:51.744201 20888 raft_consensus.cc:504] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.744244 20556 raft_consensus.cc:2454] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74f8efca8394f26b71e5f516a9b2f82 in term 0.
I20260924 10:38:51.744565 20888 leader_election.cc:288] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [CANDIDATE]: Term 1 election: Requested vote from peers f00c5337042647c8ba05b2a65a19ca8b (127.2.193.131:34733), bd621870d1c044c1830d4b47886b2650 (127.2.193.129:35411)
I20260924 10:38:51.744944 20818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721f1e7c4c3c4a8da1feaf7d6b123495" candidate_uuid: "a74f8efca8394f26b71e5f516a9b2f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00c5337042647c8ba05b2a65a19ca8b"
I20260924 10:38:51.745078 20818 raft_consensus.cc:3037] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:51.745252 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721f1e7c4c3c4a8da1feaf7d6b123495" candidate_uuid: "a74f8efca8394f26b71e5f516a9b2f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd621870d1c044c1830d4b47886b2650"
I20260924 10:38:51.745404 20556 raft_consensus.cc:3037] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:51.746479 20818 raft_consensus.cc:2454] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74f8efca8394f26b71e5f516a9b2f82 in term 1.
I20260924 10:38:51.746842 20620 leader_election.cc:302] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74f8efca8394f26b71e5f516a9b2f82, f00c5337042647c8ba05b2a65a19ca8b; no voters:
I20260924 10:38:51.746965 20556 raft_consensus.cc:2454] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74f8efca8394f26b71e5f516a9b2f82 in term 1.
I20260924 10:38:51.747031 20888 raft_consensus.cc:2781] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:51.747260 20888 raft_consensus.cc:686] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 1 LEADER]: Becoming Leader. State: Replica: a74f8efca8394f26b71e5f516a9b2f82, State: Running, Role: LEADER
I20260924 10:38:51.747617 20888 consensus_queue.cc:237] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } }
I20260924 10:38:51.751156 20413 catalog_manager.cc:5495] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74f8efca8394f26b71e5f516a9b2f82 (127.2.193.130). New cstate: current_term: 1 leader_uuid: "a74f8efca8394f26b71e5f516a9b2f82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74f8efca8394f26b71e5f516a9b2f82" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 32811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 } health_report { overall_health: UNKNOWN } } }
W20260924 10:38:51.757040 20865 tablet.cc:2369] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:51.782758 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.779421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.2.193.130:32811)
0924 10:38:51.779544 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:51.779548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:51.779562 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:38:51.779640 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:51.779642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:51.779697 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:51.779792 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:51.780006 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.780530 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.781417 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:51.781653 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:51.781736 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:51.781986 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:51.782422 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:38:51.782542 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:51.782596 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 10:38:51.782643 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:38:51.789072 20556 raft_consensus.cc:1261] T 721f1e7c4c3c4a8da1feaf7d6b123495 P bd621870d1c044c1830d4b47886b2650 [term 1 FOLLOWER]: Refusing update from remote peer a74f8efca8394f26b71e5f516a9b2f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:51.789073 20818 raft_consensus.cc:1261] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 1 FOLLOWER]: Refusing update from remote peer a74f8efca8394f26b71e5f516a9b2f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:51.789849 20888 consensus_queue.cc:1035] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd621870d1c044c1830d4b47886b2650" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:51.790093 20891 consensus_queue.cc:1035] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00c5337042647c8ba05b2a65a19ca8b" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:51.798463 20894 mvcc.cc:205] Tried to move back new op lower bound from 7332848974999928832 to 7332848974841040896. Current Snapshot: MvccSnapshot[applied={T|T < 7332848974999928832}]
I20260924 10:38:51.805812 20413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:51.805999 20413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:51.808363 20413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 957cab28948342708c656b918a1769b6: Sending DeleteTablet for 3 replicas of tablet 721f1e7c4c3c4a8da1feaf7d6b123495
I20260924 10:38:51.809209 20667 tablet_service.cc:1514] Processing DeleteTablet for tablet 721f1e7c4c3c4a8da1feaf7d6b123495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:51 UTC) from {username='slave'} at 127.0.0.1:60034
I20260924 10:38:51.809255 20536 tablet_service.cc:1514] Processing DeleteTablet for tablet 721f1e7c4c3c4a8da1feaf7d6b123495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:51 UTC) from {username='slave'} at 127.0.0.1:43372
I20260924 10:38:51.809247 20798 tablet_service.cc:1514] Processing DeleteTablet for tablet 721f1e7c4c3c4a8da1feaf7d6b123495 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:51 UTC) from {username='slave'} at 127.0.0.1:58546
I20260924 10:38:51.809780 20904 tablet_replica.cc:330] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: stopping tablet replica
I20260924 10:38:51.809880 20903 tablet_replica.cc:330] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: stopping tablet replica
I20260924 10:38:51.810138 20904 raft_consensus.cc:2229] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:38:51.810220 20903 raft_consensus.cc:2229] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:51.810494 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 20474
I20260924 10:38:51.810537 20903 raft_consensus.cc:2258] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:51.812017 20904 raft_consensus.cc:2258] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:51.812863 20903 ts_tablet_manager.cc:1905] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:51.813840 20904 ts_tablet_manager.cc:1905] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:51.815167 20903 ts_tablet_manager.cc:1918] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:51.815285 20903 log.cc:1196] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/721f1e7c4c3c4a8da1feaf7d6b123495
I20260924 10:38:51.815641 20903 ts_tablet_manager.cc:1939] T 721f1e7c4c3c4a8da1feaf7d6b123495 P f00c5337042647c8ba05b2a65a19ca8b: Deleting consensus metadata
I20260924 10:38:51.816807 20398 catalog_manager.cc:4841] TS f00c5337042647c8ba05b2a65a19ca8b (127.2.193.131:34733): tablet 721f1e7c4c3c4a8da1feaf7d6b123495 (table test-table [id=54c044e5032947859d152759b6c97a5c]) successfully deleted
I20260924 10:38:51.817268 20904 ts_tablet_manager.cc:1918] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:51.817414 20904 log.cc:1196] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/721f1e7c4c3c4a8da1feaf7d6b123495
I20260924 10:38:51.817770 20904 ts_tablet_manager.cc:1939] T 721f1e7c4c3c4a8da1feaf7d6b123495 P a74f8efca8394f26b71e5f516a9b2f82: Deleting consensus metadata
I20260924 10:38:51.818836 20398 catalog_manager.cc:4841] TS a74f8efca8394f26b71e5f516a9b2f82 (127.2.193.130:32811): tablet 721f1e7c4c3c4a8da1feaf7d6b123495 (table test-table [id=54c044e5032947859d152759b6c97a5c]) successfully deleted
I20260924 10:38:51.821664 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 20605
W20260924 10:38:51.821624 20401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:35411 (error 108)
I20260924 10:38:51.823745 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:51.823459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35411 (local address 127.0.0.1:43376)
0924 10:38:51.823590 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:51.823669 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:38:51.823901 20401 catalog_manager.cc:4584] TS bd621870d1c044c1830d4b47886b2650 (127.2.193.129:35411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 721f1e7c4c3c4a8da1feaf7d6b123495: Network error: Client connection negotiation failed: client connection to 127.2.193.129:35411: connect: Connection refused (error 111)
I20260924 10:38:51.826799 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 20736
I20260924 10:38:51.831585 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 20383
2026-09-24T10:38:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1134 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:38:51 dist-test-slave-8wc7 krb5kdc[20909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:38:51 dist-test-slave-8wc7 krb5kdc[20909](info): set up 2 sockets
Sep 24 10:38:51 dist-test-slave-8wc7 krb5kdc[20909](info): commencing operation
krb5kdc: starting...
W20260924 10:38:52.865299 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:38:52 dist-test-slave-8wc7 krb5kdc[20909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:38:52.992664 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36639
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:36639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:53.107343 20922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:53.107723 20922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:53.107796 20922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:53.111984 20922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:53.112090 20922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:53.112128 20922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:53.112164 20922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:53.112193 20922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:53.117794 20922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:36639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:53.118959 20922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:53.120046 20922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:53.128804 20928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:53.128804 20927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:53.128983 20922 server_base.cc:1034] running on GCE node
W20260924 10:38:53.129148 20930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:53.129711 20922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:53.130882 20922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:53.132057 20922 hybrid_clock.cc:648] HybridClock initialized: now 1790246333132029 us; error 41 us; skew 500 ppm
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:53.134908 20922 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:38:53.135896 20922 webserver.cc:460] Webserver started at http://127.2.193.190:44897/ using document root <none> and password file <none>
I20260924 10:38:53.136493 20922 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:53.136591 20922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:53.136834 20922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:53.139052 20922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "a028699aae8f4d099357cc8718a4b0b3"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "efa03147abe66ac3f3ab39c3760488e4"
server_key_iv: "f892992032f0899510df9ca1346d5f27"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.139689 20922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "a028699aae8f4d099357cc8718a4b0b3"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "efa03147abe66ac3f3ab39c3760488e4"
server_key_iv: "f892992032f0899510df9ca1346d5f27"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.143726 20922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:38:53.146466 20937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.148013 20922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:38:53.148195 20922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "a028699aae8f4d099357cc8718a4b0b3"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "efa03147abe66ac3f3ab39c3760488e4"
server_key_iv: "f892992032f0899510df9ca1346d5f27"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.148337 20922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:53.167778 20922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:53.171082 20922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:53.171319 20922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:53.178892 20922 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:36639
I20260924 10:38:53.178934 20989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:36639 every 8 connection(s)
I20260924 10:38:53.179952 20922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:53.183036 20990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:53.188987 20990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3: Bootstrap starting.
I20260924 10:38:53.189903 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 20922
I20260924 10:38:53.190052 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:38:53.190343 2822 external_mini_cluster.cc:1428] Setting key c58a1b6d81cc40e9d98113e95c2ea2ce
I20260924 10:38:53.192451 20990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:53.193310 20990 log.cc:824] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:53.195696 20990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3: No bootstrap required, opened a new log
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:53.200556 20990 raft_consensus.cc:348] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a028699aae8f4d099357cc8718a4b0b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36639 } }
I20260924 10:38:53.200839 20990 raft_consensus.cc:374] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:53.200901 20990 raft_consensus.cc:729] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a028699aae8f4d099357cc8718a4b0b3, State: Initialized, Role: FOLLOWER
I20260924 10:38:53.201426 20990 consensus_queue.cc:260] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a028699aae8f4d099357cc8718a4b0b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36639 } }
I20260924 10:38:53.201632 20990 raft_consensus.cc:388] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:53.201709 20990 raft_consensus.cc:482] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:53.201819 20990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:53.202927 20990 raft_consensus.cc:504] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a028699aae8f4d099357cc8718a4b0b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36639 } }
I20260924 10:38:53.203293 20990 leader_election.cc:302] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a028699aae8f4d099357cc8718a4b0b3; no voters:
I20260924 10:38:53.203650 20990 leader_election.cc:288] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:38:53.203781 20995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:53.204027 20995 raft_consensus.cc:686] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [term 1 LEADER]: Becoming Leader. State: Replica: a028699aae8f4d099357cc8718a4b0b3, State: Running, Role: LEADER
I20260924 10:38:53.204345 20995 consensus_queue.cc:237] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a028699aae8f4d099357cc8718a4b0b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36639 } }
I20260924 10:38:53.205073 20990 sys_catalog.cc:564] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:53.206203 20996 sys_catalog.cc:455] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a028699aae8f4d099357cc8718a4b0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a028699aae8f4d099357cc8718a4b0b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36639 } } }
I20260924 10:38:53.206331 20996 sys_catalog.cc:458] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:53.206774 20997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a028699aae8f4d099357cc8718a4b0b3. Latest consensus state: current_term: 1 leader_uuid: "a028699aae8f4d099357cc8718a4b0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a028699aae8f4d099357cc8718a4b0b3" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36639 } } }
I20260924 10:38:53.206799 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.191886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.2.193.190:36639)
0924 10:38:53.192362 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.192378 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.192422 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:38:53.193269 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.193291 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.193743 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.194161 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:53.195062 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.196176 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.197458 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.197955 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.198088 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.201121 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:53.201127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:53.201141 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:53.201220 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:53.203167 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.203710 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.203717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.203724 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.203884 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.204243 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.204246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.204248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.204778 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:53.204985 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.205345 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.205675 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260924 10:38:53.206960 20997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3 [sys.catalog]: This master's current role is: LEADER
W20260924 10:38:53.210695 21010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:38:53.210795 21010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:38:53.210891 21004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:53.211637 21004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:53.217634 21004 catalog_manager.cc:1347] Generated new cluster ID: c3a8058a327f42749d7b57c360948b8c
I20260924 10:38:53.217720 21004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:53.235883 21004 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:38:53.237162 21004 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:53.248289 21004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3: Generated new TSK 0
I20260924 10:38:53.249063 21004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:38:53.285673 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36639
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:53.403200 21016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:53.403627 21016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:53.403712 21016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:53.410732 21016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:53.410801 21016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:53.410868 21016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:53.416546 21016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:53.417694 21016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:53.418699 21016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:53.428784 21024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:53.428939 21021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:53.428970 21022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:53.429185 21016 server_base.cc:1034] running on GCE node
I20260924 10:38:53.429654 21016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:53.430341 21016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:53.431510 21016 hybrid_clock.cc:648] HybridClock initialized: now 1790246333431498 us; error 25 us; skew 500 ppm
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:53.440286 21016 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:38:53.441365 21016 webserver.cc:460] Webserver started at http://127.2.193.129:44237/ using document root <none> and password file <none>
I20260924 10:38:53.441941 21016 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:53.442013 21016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:53.442188 21016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:53.444332 21016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "5053f27c2746433c9aaf4bb4c93f5913"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "4bf4c3eb3901346a632c52aadf31e322"
server_key_iv: "5b7f3563267e4ac7158cd908e82668ef"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.444871 21016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "5053f27c2746433c9aaf4bb4c93f5913"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "4bf4c3eb3901346a632c52aadf31e322"
server_key_iv: "5b7f3563267e4ac7158cd908e82668ef"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.448808 21016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 10:38:53.451443 21031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.452869 21016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:38:53.453022 21016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "5053f27c2746433c9aaf4bb4c93f5913"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "4bf4c3eb3901346a632c52aadf31e322"
server_key_iv: "5b7f3563267e4ac7158cd908e82668ef"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.453172 21016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:53.475014 21016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:53.478434 21016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:53.478669 21016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:53.479324 21016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:53.480206 21016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:53.480283 21016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.480351 21016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:53.480400 21016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.489962 21016 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:45371
I20260924 10:38:53.489979 21144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:45371 every 8 connection(s)
I20260924 10:38:53.490917 21016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:53.492960 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21016
I20260924 10:38:53.493085 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:38:53.493443 2822 external_mini_cluster.cc:1428] Setting key 61dee9c1132b1e4049067880f51bc908
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:38:53.507136 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.492807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:34783 (local address 127.2.193.190:36639)
0924 10:38:53.492946 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.492950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.494228 (+ 1278us) server_negotiation.cc:407] Connection header received
0924 10:38:53.495842 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.495845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.495900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.495988 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:53.497703 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.498252 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.499580 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.499837 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.499963 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.502798 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:53.502802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:53.502804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:53.502832 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:53.504393 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.505032 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.505035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.505036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.505129 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.505596 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.505599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.505600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.505796 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:53.505875 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.506748 (+ 873us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.506911 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:38:53.508159 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.493403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36639 (local address 127.2.193.129:34783)
0924 10:38:53.494048 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:53.494112 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:53.495612 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:53.496172 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:53.496180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:53.496680 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:53.497488 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.497514 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.498395 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.498398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.499418 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.499429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.500009 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.500013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.500198 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.500910 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:53.500928 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:53.502592 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:53.504530 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:53.504536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:53.504554 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:53.504853 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:53.505305 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:53.505308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:53.505309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:53.505453 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:53.505942 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:53.505952 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:53.506266 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 10:38:53.506586 (+ 320us) client_negotiation.cc:240] Negotiation successful
0924 10:38:53.506924 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":150,"threads_started":1}
I20260924 10:38:53.509601 21145 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36639
I20260924 10:38:53.509871 21145 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:53.510401 21145 heartbeater.cc:511] Master 127.2.193.190:36639 requested a full tablet report, sending...
I20260924 10:38:53.511991 20954 ts_manager.cc:194] Registered new tserver with Master: 5053f27c2746433c9aaf4bb4c93f5913 (127.2.193.129:45371)
I20260924 10:38:53.513154 20954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:34783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:38:53.522717 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36639
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:53.636749 21150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:53.637042 21150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:53.637095 21150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:53.641405 21150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:53.641477 21150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:53.641566 21150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:53.647238 21150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:53.648452 21150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:53.649585 21150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:53.659932 21158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:53.660040 21150 server_base.cc:1034] running on GCE node
W20260924 10:38:53.659936 21155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:53.659953 21156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:53.660715 21150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:53.661414 21150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:53.662573 21150 hybrid_clock.cc:648] HybridClock initialized: now 1790246333662537 us; error 48 us; skew 500 ppm
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:53.665477 21150 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:38:53.666425 21150 webserver.cc:460] Webserver started at http://127.2.193.130:35383/ using document root <none> and password file <none>
I20260924 10:38:53.667014 21150 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:53.667084 21150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:53.667308 21150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:53.669538 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "d993670ec97e478bbdcf31ed6ae9ce11"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "8bbba91ee7e1da7ba2d12f94769888c0"
server_key_iv: "c561fc00704bb962d8edb930dafd753c"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.670126 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "d993670ec97e478bbdcf31ed6ae9ce11"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "8bbba91ee7e1da7ba2d12f94769888c0"
server_key_iv: "c561fc00704bb962d8edb930dafd753c"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.673956 21150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:38:53.676355 21165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.677677 21150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:38:53.677809 21150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "d993670ec97e478bbdcf31ed6ae9ce11"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "8bbba91ee7e1da7ba2d12f94769888c0"
server_key_iv: "c561fc00704bb962d8edb930dafd753c"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.677922 21150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:53.708986 21150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:53.712077 21150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:53.712292 21150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:53.712886 21150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:53.713740 21150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:53.713814 21150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.713881 21150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:53.713932 21150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.722635 21150 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:39543
I20260924 10:38:53.722661 21278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:39543 every 8 connection(s)
I20260924 10:38:53.723510 21150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:53.729995 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21150
I20260924 10:38:53.730135 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:38:53.730401 2822 external_mini_cluster.cc:1428] Setting key a1918334cdcbf05188fb05be5cb2a2ea
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:53.737802 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.725298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:34175 (local address 127.2.193.190:36639)
0924 10:38:53.725463 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.725469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.726426 (+ 957us) server_negotiation.cc:407] Connection header received
0924 10:38:53.727686 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.727689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.727736 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.727811 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:53.728910 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.729500 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.730779 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.731047 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.731160 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.733922 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:53.733927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:53.733932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:53.733964 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:53.735483 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.736092 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.736094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.736095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.736161 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.736580 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.736583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.736604 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.736800 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:53.736876 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.737582 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.737678 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:38:53.739095 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.725808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36639 (local address 127.2.193.130:34175)
0924 10:38:53.726277 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:53.726318 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:53.727487 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:53.727982 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:53.727988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:53.728326 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:53.728790 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.728802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.729606 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.729608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.730659 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.730668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.731226 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.731229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.731413 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.732065 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:53.732080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:53.733755 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:53.735601 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:53.735610 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:53.735628 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:53.735995 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:53.736252 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:53.736258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:53.736263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:53.736456 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:53.736905 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:53.736915 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:53.737281 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 10:38:53.737539 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 10:38:53.737931 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":134,"threads_started":1}
I20260924 10:38:53.740319 21279 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36639
I20260924 10:38:53.740600 21279 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:38:53.741200 21279 heartbeater.cc:511] Master 127.2.193.190:36639 requested a full tablet report, sending...
I20260924 10:38:53.742278 20954 ts_manager.cc:194] Registered new tserver with Master: d993670ec97e478bbdcf31ed6ae9ce11 (127.2.193.130:39543)
I20260924 10:38:53.742843 20954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:34175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:38:53.759770 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36639
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:53.871551 21284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:53.871832 21284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:53.871877 21284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:53.876137 21284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:53.876188 21284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:53.876248 21284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:53.881728 21284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:53.882795 21284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:53.883839 21284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:53.893865 21289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:53.893877 21290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:53.893877 21292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:53.894238 21284 server_base.cc:1034] running on GCE node
I20260924 10:38:53.894644 21284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:53.895243 21284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:53.896404 21284 hybrid_clock.cc:648] HybridClock initialized: now 1790246333896392 us; error 29 us; skew 500 ppm
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:53.899266 21284 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:38:53.900208 21284 webserver.cc:460] Webserver started at http://127.2.193.131:43569/ using document root <none> and password file <none>
I20260924 10:38:53.900785 21284 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:53.900878 21284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:53.901113 21284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:53.903260 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "f3116f2c7a6f432ca2b73777b2f83f40"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "715917a0d7f88e664ef74e9badb26605"
server_key_iv: "8839e05af720bb0bf9a6d5ebdc3b3ed4"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.903853 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "f3116f2c7a6f432ca2b73777b2f83f40"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "715917a0d7f88e664ef74e9badb26605"
server_key_iv: "8839e05af720bb0bf9a6d5ebdc3b3ed4"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.908135 21284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:38:53.910594 21299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.911839 21284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:38:53.911993 21284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "f3116f2c7a6f432ca2b73777b2f83f40"
format_stamp: "Formatted at 2026-09-24 10:38:53 on dist-test-slave-8wc7"
server_key: "715917a0d7f88e664ef74e9badb26605"
server_key_iv: "8839e05af720bb0bf9a6d5ebdc3b3ed4"
server_key_version: "encryptionkey@0"
I20260924 10:38:53.912110 21284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:53.924464 21284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:53.927852 21284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:53.928074 21284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:53.928692 21284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:53.929669 21284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:53.929718 21284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.929795 21284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:53.929828 21284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:53.938645 21284 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:36299
I20260924 10:38:53.938732 21412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:36299 every 8 connection(s)
I20260924 10:38:53.939661 21284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:53.946727 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21284
I20260924 10:38:53.946841 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:38:53.947175 2822 external_mini_cluster.cc:1428] Setting key 5b733d8afdd2a44c64dd64b187984c2f
Sep 24 10:38:53 dist-test-slave-8wc7 krb5kdc[20909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246333, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:53.954685 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.941695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:47259 (local address 127.2.193.190:36639)
0924 10:38:53.941839 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.941842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.942758 (+ 916us) server_negotiation.cc:407] Connection header received
0924 10:38:53.944043 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.944046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.944097 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.944185 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:53.945705 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.946251 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.947625 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.947945 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.948040 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.950929 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:53.950934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:53.950937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:53.950969 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:53.952466 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.953028 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.953032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.953034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.953127 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.953555 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.953558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.953560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.953763 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:53.953870 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.954389 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.954515 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:38:53.955549 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.942138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:36639 (local address 127.2.193.131:47259)
0924 10:38:53.942590 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:53.942635 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:53.943820 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:53.944382 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:53.944389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:53.944770 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:53.945507 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.945535 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.946367 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.946370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.947475 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.947486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.948116 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.948119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.948337 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.949047 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:53.949064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:53.950728 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:53.952591 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:53.952615 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:53.952629 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:53.952896 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:53.953265 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:53.953268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:53.953271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:53.953418 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:53.953882 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:53.953891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:53.954147 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 10:38:53.954351 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:38:53.954706 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260924 10:38:53.956765 21413 heartbeater.cc:347] Connected to a master server at 127.2.193.190:36639
I20260924 10:38:53.957056 21413 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:53.957646 21413 heartbeater.cc:511] Master 127.2.193.190:36639 requested a full tablet report, sending...
I20260924 10:38:53.958870 20954 ts_manager.cc:194] Registered new tserver with Master: f3116f2c7a6f432ca2b73777b2f83f40 (127.2.193.131:36299)
I20260924 10:38:53.959476 20954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:47259
I20260924 10:38:53.961459 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:38:53.971427 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.963601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.2.193.190:36639)
0924 10:38:53.963754 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.963759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.963881 (+ 122us) server_negotiation.cc:407] Connection header received
0924 10:38:53.963979 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.963981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.964030 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.964098 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:53.964734 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.965325 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.966194 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.966416 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.966525 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.968187 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:53.968191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:53.968195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:53.968222 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:53.969563 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.970092 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.970096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.970098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.970175 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:53.970507 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:53.970511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:53.970512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:53.970723 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:53.970887 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.971164 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.971259 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:38:53.974555 20954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:38:53.977715 20954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:38:53.992826 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.986432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.2.193.131:36299)
0924 10:38:53.986793 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.986807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.987235 (+ 428us) server_negotiation.cc:407] Connection header received
0924 10:38:53.987551 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.987557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.987787 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.987921 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:53.988486 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.989729 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.991478 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.992139 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.992262 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.992405 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.992546 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.992662 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":138,"threads_started":1}
I20260924 10:38:53.992816 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.986515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:39543 (local address 127.0.0.1:59672)
0924 10:38:53.987513 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:53.987540 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:53.987624 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:53.988306 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:53.988309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:53.988325 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:53.988424 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.988436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.989885 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.989888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.991208 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.991218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.992317 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.992321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.992426 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.992447 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:38:53.992510 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:38:53.992664 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":107,"thread_start_us":154,"threads_started":1}
I20260924 10:38:53.992833 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.987442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.2.193.130:39543)
0924 10:38:53.987861 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.987876 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.987896 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:38:53.987968 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.987976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.988172 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.988324 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:53.988525 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.989729 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.991341 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.992190 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.992366 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.992512 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.992576 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.992664 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":147,"threads_started":1}
I20260924 10:38:53.993009 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.987022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:45371 (local address 127.0.0.1:54086)
0924 10:38:53.987962 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:53.987972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:53.988092 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:53.988575 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:53.988577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:53.988594 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:53.988668 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.988676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.989707 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.989711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.990948 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.990958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.992197 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.992200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.992320 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.992718 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 10:38:53.992784 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 10:38:53.992850 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":6,"thread_start_us":86,"threads_started":1}
I20260924 10:38:53.993701 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.987201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.2.193.129:45371)
0924 10:38:53.987854 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:53.987867 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:53.988076 (+ 209us) server_negotiation.cc:407] Connection header received
0924 10:38:53.988206 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:53.988211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:53.988405 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:53.988556 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:53.988760 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.989569 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.991087 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.991856 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.991964 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.992128 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:53.993265 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0924 10:38:53.993370 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":94,"threads_started":1}
I20260924 10:38:53.996196 21079 tablet_service.cc:1467] Processing CreateTablet for tablet f3289e89edb64d6e86e2de3cbd37bc1c (DEFAULT_TABLE table=test-table [id=556255130868424b900cc67ed4fd4e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:53.996196 21213 tablet_service.cc:1467] Processing CreateTablet for tablet f3289e89edb64d6e86e2de3cbd37bc1c (DEFAULT_TABLE table=test-table [id=556255130868424b900cc67ed4fd4e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:53.993077 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:53.986248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:36299 (local address 127.0.0.1:43398)
0924 10:38:53.987070 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:53.987126 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:53.987347 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:53.988156 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:53.988162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:53.988185 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:53.988339 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.988345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.989885 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.989888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.991355 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:53.991363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:53.992262 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:53.992265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:53.992359 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:53.992406 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 10:38:53.992515 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:38:53.992670 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":293,"thread_start_us":110,"threads_started":1}
I20260924 10:38:53.997455 21079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3289e89edb64d6e86e2de3cbd37bc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:53.997455 21213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3289e89edb64d6e86e2de3cbd37bc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:53.999637 21347 tablet_service.cc:1467] Processing CreateTablet for tablet f3289e89edb64d6e86e2de3cbd37bc1c (DEFAULT_TABLE table=test-table [id=556255130868424b900cc67ed4fd4e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:54.000566 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3289e89edb64d6e86e2de3cbd37bc1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:54.005086 21431 tablet_bootstrap.cc:492] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913: Bootstrap starting.
I20260924 10:38:54.005565 21430 tablet_bootstrap.cc:492] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: Bootstrap starting.
I20260924 10:38:54.007494 21432 tablet_bootstrap.cc:492] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Bootstrap starting.
I20260924 10:38:54.008320 21431 tablet_bootstrap.cc:654] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:54.008822 21430 tablet_bootstrap.cc:654] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:54.009593 21432 tablet_bootstrap.cc:654] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:54.009615 21430 log.cc:824] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:54.009729 21431 log.cc:824] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:54.010743 21432 log.cc:824] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:54.011663 21431 tablet_bootstrap.cc:492] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913: No bootstrap required, opened a new log
I20260924 10:38:54.011694 21430 tablet_bootstrap.cc:492] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: No bootstrap required, opened a new log
I20260924 10:38:54.011855 21431 ts_tablet_manager.cc:1397] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:38:54.011869 21430 ts_tablet_manager.cc:1397] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:38:54.012887 21432 tablet_bootstrap.cc:492] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: No bootstrap required, opened a new log
I20260924 10:38:54.013073 21432 ts_tablet_manager.cc:1397] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:38:54.014945 21430 raft_consensus.cc:348] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.015277 21430 raft_consensus.cc:374] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:54.015336 21430 raft_consensus.cc:729] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d993670ec97e478bbdcf31ed6ae9ce11, State: Initialized, Role: FOLLOWER
I20260924 10:38:54.015781 21430 consensus_queue.cc:260] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.015792 21431 raft_consensus.cc:348] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.016036 21431 raft_consensus.cc:374] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:54.016086 21431 raft_consensus.cc:729] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5053f27c2746433c9aaf4bb4c93f5913, State: Initialized, Role: FOLLOWER
I20260924 10:38:54.016628 21279 heartbeater.cc:503] Master 127.2.193.190:36639 was elected leader, sending a full tablet report...
I20260924 10:38:54.016557 21431 consensus_queue.cc:260] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.016878 21430 ts_tablet_manager.cc:1428] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:38:54.017544 21431 ts_tablet_manager.cc:1428] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:38:54.017437 21432 raft_consensus.cc:348] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.017669 21432 raft_consensus.cc:374] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:54.017721 21432 raft_consensus.cc:729] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3116f2c7a6f432ca2b73777b2f83f40, State: Initialized, Role: FOLLOWER
I20260924 10:38:54.017729 21145 heartbeater.cc:503] Master 127.2.193.190:36639 was elected leader, sending a full tablet report...
I20260924 10:38:54.018126 21432 consensus_queue.cc:260] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.018762 21413 heartbeater.cc:503] Master 127.2.193.190:36639 was elected leader, sending a full tablet report...
I20260924 10:38:54.019905 21432 ts_tablet_manager.cc:1428] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:38:54.028986 21436 raft_consensus.cc:482] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:54.029173 21436 raft_consensus.cc:504] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.030252 21436 leader_election.cc:288] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3116f2c7a6f432ca2b73777b2f83f40 (127.2.193.131:36299), 5053f27c2746433c9aaf4bb4c93f5913 (127.2.193.129:45371)
I20260924 10:38:54.034761 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.030601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:42319 (local address 127.2.193.129:45371)
0924 10:38:54.030744 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.030749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.030767 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:38:54.030822 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.030825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.030883 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.030971 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:54.031257 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.032060 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.033545 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.034148 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.034270 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.034339 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.034562 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.034633 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:38:54.034761 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.030665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:49485 (local address 127.2.193.131:36299)
0924 10:38:54.030786 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.030791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.031182 (+ 391us) server_negotiation.cc:407] Connection header received
0924 10:38:54.031293 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.031296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.031341 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.031418 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:54.031648 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.032298 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.033661 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.034259 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.034359 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.034403 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.034565 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.034633 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:38:54.034824 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.030477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:45371 (local address 127.2.193.130:42319)
0924 10:38:54.030623 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.030641 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.030734 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.030989 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.030992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.031011 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:54.031143 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.031153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.032163 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.032165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.033427 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.033438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.034267 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.034269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.034408 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.034438 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.034546 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.034667 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":38}
I20260924 10:38:54.035050 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.030477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:36299 (local address 127.2.193.130:49485)
0924 10:38:54.031084 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.031098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.031208 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.031432 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.031434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.031446 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:54.031549 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.031553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.032390 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.032393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.033565 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.033572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.034354 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.034361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.034446 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.034470 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.034537 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.034667 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":254,"thread_start_us":129,"threads_started":1}
I20260924 10:38:54.035522 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3289e89edb64d6e86e2de3cbd37bc1c" candidate_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5053f27c2746433c9aaf4bb4c93f5913" is_pre_election: true
I20260924 10:38:54.035559 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3289e89edb64d6e86e2de3cbd37bc1c" candidate_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" is_pre_election: true
I20260924 10:38:54.035851 21099 raft_consensus.cc:2454] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d993670ec97e478bbdcf31ed6ae9ce11 in term 0.
I20260924 10:38:54.035849 21367 raft_consensus.cc:2454] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d993670ec97e478bbdcf31ed6ae9ce11 in term 0.
I20260924 10:38:54.036340 21167 leader_election.cc:302] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5053f27c2746433c9aaf4bb4c93f5913, d993670ec97e478bbdcf31ed6ae9ce11, f3116f2c7a6f432ca2b73777b2f83f40; no voters:
I20260924 10:38:54.036595 21436 raft_consensus.cc:2781] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:38:54.036676 21436 raft_consensus.cc:482] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:38:54.036725 21436 raft_consensus.cc:3037] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:54.037845 21436 raft_consensus.cc:504] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.038223 21436 leader_election.cc:288] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [CANDIDATE]: Term 1 election: Requested vote from peers f3116f2c7a6f432ca2b73777b2f83f40 (127.2.193.131:36299), 5053f27c2746433c9aaf4bb4c93f5913 (127.2.193.129:45371)
I20260924 10:38:54.038578 21099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3289e89edb64d6e86e2de3cbd37bc1c" candidate_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5053f27c2746433c9aaf4bb4c93f5913"
I20260924 10:38:54.038580 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3289e89edb64d6e86e2de3cbd37bc1c" candidate_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3116f2c7a6f432ca2b73777b2f83f40"
I20260924 10:38:54.038718 21099 raft_consensus.cc:3037] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:54.038743 21367 raft_consensus.cc:3037] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:54.039810 21367 raft_consensus.cc:2454] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d993670ec97e478bbdcf31ed6ae9ce11 in term 1.
I20260924 10:38:54.039819 21099 raft_consensus.cc:2454] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d993670ec97e478bbdcf31ed6ae9ce11 in term 1.
I20260924 10:38:54.040155 21166 leader_election.cc:302] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5053f27c2746433c9aaf4bb4c93f5913, d993670ec97e478bbdcf31ed6ae9ce11; no voters:
I20260924 10:38:54.040360 21436 raft_consensus.cc:2781] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:54.040624 21436 raft_consensus.cc:686] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 1 LEADER]: Becoming Leader. State: Replica: d993670ec97e478bbdcf31ed6ae9ce11, State: Running, Role: LEADER
I20260924 10:38:54.040969 21436 consensus_queue.cc:237] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } }
I20260924 10:38:54.044426 20953 catalog_manager.cc:5495] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 reported cstate change: term changed from 0 to 1, leader changed from <none> to d993670ec97e478bbdcf31ed6ae9ce11 (127.2.193.130). New cstate: current_term: 1 leader_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d993670ec97e478bbdcf31ed6ae9ce11" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 39543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 } health_report { overall_health: UNKNOWN } } }
I20260924 10:38:54.061558 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.058244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.2.193.130:39543)
0924 10:38:54.058389 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.058393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.058442 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:38:54.058518 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.058520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.058574 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.058672 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:54.058885 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.059384 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.060162 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.060363 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.060435 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.060720 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:54.061137 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:38:54.061332 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.061391 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.061426 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:38:54.067849 21099 raft_consensus.cc:1261] T f3289e89edb64d6e86e2de3cbd37bc1c P 5053f27c2746433c9aaf4bb4c93f5913 [term 1 FOLLOWER]: Refusing update from remote peer d993670ec97e478bbdcf31ed6ae9ce11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:54.067849 21367 raft_consensus.cc:1261] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 1 FOLLOWER]: Refusing update from remote peer d993670ec97e478bbdcf31ed6ae9ce11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:54.068584 21440 consensus_queue.cc:1035] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3116f2c7a6f432ca2b73777b2f83f40" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:54.068847 21436 consensus_queue.cc:1035] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5053f27c2746433c9aaf4bb4c93f5913" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 45371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:54.076298 21443 mvcc.cc:205] Tried to move back new op lower bound from 7332848984334118912 to 7332848984234905600. Current Snapshot: MvccSnapshot[applied={T|T < 7332848984334118912}]
I20260924 10:38:54.076721 21444 mvcc.cc:205] Tried to move back new op lower bound from 7332848984334118912 to 7332848984234905600. Current Snapshot: MvccSnapshot[applied={T|T < 7332848984334118912}]
I20260924 10:38:54.077339 21445 mvcc.cc:205] Tried to move back new op lower bound from 7332848984334118912 to 7332848984234905600. Current Snapshot: MvccSnapshot[applied={T|T < 7332848984334118912}]
I20260924 10:38:54.085577 20952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:54.085743 20952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:54.088325 20952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a028699aae8f4d099357cc8718a4b0b3: Sending DeleteTablet for 3 replicas of tablet f3289e89edb64d6e86e2de3cbd37bc1c
I20260924 10:38:54.089216 21213 tablet_service.cc:1514] Processing DeleteTablet for tablet f3289e89edb64d6e86e2de3cbd37bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:54 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:59672
I20260924 10:38:54.089354 21079 tablet_service.cc:1514] Processing DeleteTablet for tablet f3289e89edb64d6e86e2de3cbd37bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:54 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:54086
I20260924 10:38:54.089674 21347 tablet_service.cc:1514] Processing DeleteTablet for tablet f3289e89edb64d6e86e2de3cbd37bc1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:54 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:43398
I20260924 10:38:54.090029 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21016
I20260924 10:38:54.090746 21452 tablet_replica.cc:330] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: stopping tablet replica
I20260924 10:38:54.091135 21452 raft_consensus.cc:2229] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:38:54.091220 21454 tablet_replica.cc:330] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: stopping tablet replica
I20260924 10:38:54.091512 21454 raft_consensus.cc:2229] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:54.091699 21452 raft_consensus.cc:2258] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:54.091837 21454 raft_consensus.cc:2258] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:54.093611 21454 ts_tablet_manager.cc:1905] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:54.094877 21452 ts_tablet_manager.cc:1905] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:54.096423 21454 ts_tablet_manager.cc:1918] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:54.096544 21454 log.cc:1196] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/f3289e89edb64d6e86e2de3cbd37bc1c
I20260924 10:38:54.096908 21454 ts_tablet_manager.cc:1939] T f3289e89edb64d6e86e2de3cbd37bc1c P f3116f2c7a6f432ca2b73777b2f83f40: Deleting consensus metadata
W20260924 10:38:54.096912 20938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:45371 (error 108)
I20260924 10:38:54.097745 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21150
I20260924 10:38:54.097951 21452 ts_tablet_manager.cc:1918] T f3289e89edb64d6e86e2de3cbd37bc1c P d993670ec97e478bbdcf31ed6ae9ce11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:54.098788 20939 catalog_manager.cc:4841] TS f3116f2c7a6f432ca2b73777b2f83f40 (127.2.193.131:36299): tablet f3289e89edb64d6e86e2de3cbd37bc1c (table test-table [id=556255130868424b900cc67ed4fd4e77]) successfully deleted
I20260924 10:38:54.099157 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.098760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:45371 (local address 127.0.0.1:54094)
0924 10:38:54.098957 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.099044 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 10:38:54.099336 20938 catalog_manager.cc:4584] TS 5053f27c2746433c9aaf4bb4c93f5913 (127.2.193.129:45371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3289e89edb64d6e86e2de3cbd37bc1c: Network error: Client connection negotiation failed: client connection to 127.2.193.129:45371: connect: Connection refused (error 111)
I20260924 10:38:54.104351 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21284
I20260924 10:38:54.105324 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.104431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:39543 (local address 127.0.0.1:59692)
0924 10:38:54.105134 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.105218 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606}
I20260924 10:38:54.110064 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 20922
2026-09-24T10:38:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2278 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T10:38:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:38:54Z Disabled control of system clock
I20260924 10:38:54.139760 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46501
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:38:54.254786 21459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:54.255054 21459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:54.255090 21459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:54.259150 21459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:54.259208 21459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:54.259223 21459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:54.259240 21459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:54.259256 21459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:54.264092 21459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:54.265139 21459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:54.266678 21459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:54.275386 21467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:54.275533 21459 server_base.cc:1034] running on GCE node
W20260924 10:38:54.275406 21465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:54.275394 21464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:54.276314 21459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:54.277577 21459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:54.278800 21459 hybrid_clock.cc:648] HybridClock initialized: now 1790246334278786 us; error 28 us; skew 500 ppm
I20260924 10:38:54.280781 21459 webserver.cc:460] Webserver started at http://127.2.193.190:33717/ using document root <none> and password file <none>
I20260924 10:38:54.281452 21459 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:54.281556 21459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:54.281796 21459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:54.283887 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "5e74eec9cb1e4ad799125dc6b4a46db7"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "4f03447c38e1c7c5fa377f3d2e0b78ba"
server_key_iv: "366a11f9900d69e5054deb56770060f0"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.284461 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "5e74eec9cb1e4ad799125dc6b4a46db7"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "4f03447c38e1c7c5fa377f3d2e0b78ba"
server_key_iv: "366a11f9900d69e5054deb56770060f0"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.288235 21459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:38:54.290758 21473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.292025 21459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:38:54.292167 21459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "5e74eec9cb1e4ad799125dc6b4a46db7"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "4f03447c38e1c7c5fa377f3d2e0b78ba"
server_key_iv: "366a11f9900d69e5054deb56770060f0"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.292285 21459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:54.307488 21459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:54.308184 21459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:54.308390 21459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:54.315244 21459 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:46501
I20260924 10:38:54.315249 21525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:46501 every 8 connection(s)
I20260924 10:38:54.316179 21459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:54.319301 21526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:54.324723 21526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7: Bootstrap starting.
I20260924 10:38:54.326002 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21459
I20260924 10:38:54.326138 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:38:54.326397 2822 external_mini_cluster.cc:1428] Setting key 65296e5612cbedefd01d551704215290
I20260924 10:38:54.327497 21526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:54.328320 21526 log.cc:824] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:54.330385 21526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7: No bootstrap required, opened a new log
I20260924 10:38:54.333987 21526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46501 } }
I20260924 10:38:54.334244 21526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:54.334331 21526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e74eec9cb1e4ad799125dc6b4a46db7, State: Initialized, Role: FOLLOWER
I20260924 10:38:54.334860 21526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46501 } }
I20260924 10:38:54.335024 21526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:54.335114 21526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:54.335237 21526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:54.335799 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.327504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.2.193.190:46501)
0924 10:38:54.328151 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.328165 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.328198 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:38:54.329007 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.329033 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.329441 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.329795 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:54.330227 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.331313 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.332251 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.332723 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.332893 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.333789 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:54.333794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:54.333807 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:54.334204 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:54.334292 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:54.334320 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:54.334418 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.334608 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.334895 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260924 10:38:54.336335 21526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46501 } }
I20260924 10:38:54.336699 21526 leader_election.cc:302] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e74eec9cb1e4ad799125dc6b4a46db7; no voters:
I20260924 10:38:54.337052 21526 leader_election.cc:288] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:38:54.337134 21531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:54.337337 21531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e74eec9cb1e4ad799125dc6b4a46db7, State: Running, Role: LEADER
I20260924 10:38:54.337630 21531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46501 } }
I20260924 10:38:54.338352 21526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:54.338838 21531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e74eec9cb1e4ad799125dc6b4a46db7. Latest consensus state: current_term: 1 leader_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46501 } } }
I20260924 10:38:54.338954 21531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:54.339355 21532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e74eec9cb1e4ad799125dc6b4a46db7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46501 } } }
I20260924 10:38:54.339517 21532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:54.340577 21535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:54.343495 21535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:54.348649 21535 catalog_manager.cc:1347] Generated new cluster ID: cd677b8c00e64b12b9a7e438c5d5d545
I20260924 10:38:54.348716 21535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:54.364602 21535 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:38:54.365639 21535 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:54.374181 21535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7: Generated new TSK 0
I20260924 10:38:54.375061 21535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:38:54.384159 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46501
--builtin_ntp_servers=127.2.193.148:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:38:54.500888 21550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:54.501222 21550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:54.501281 21550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:54.505319 21550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:54.505376 21550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:54.505431 21550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:54.510310 21550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:54.511332 21550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:54.512677 21550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:54.522369 21555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:54.522588 21550 server_base.cc:1034] running on GCE node
W20260924 10:38:54.522369 21556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:54.522388 21558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:54.523105 21550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:54.523772 21550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:54.524971 21550 hybrid_clock.cc:648] HybridClock initialized: now 1790246334524928 us; error 49 us; skew 500 ppm
I20260924 10:38:54.527017 21550 webserver.cc:460] Webserver started at http://127.2.193.129:40183/ using document root <none> and password file <none>
I20260924 10:38:54.527642 21550 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:54.527738 21550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:54.527969 21550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:54.530166 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "b93268d0e4874fafbd63bb8b625c81e4"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "d63d129a679b7000e731b0477b286612"
server_key_iv: "f13ad041f7882feed1764d7565a41f7e"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.530754 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "b93268d0e4874fafbd63bb8b625c81e4"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "d63d129a679b7000e731b0477b286612"
server_key_iv: "f13ad041f7882feed1764d7565a41f7e"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.534507 21550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:38:54.536988 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.538301 21550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:38:54.538439 21550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "b93268d0e4874fafbd63bb8b625c81e4"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "d63d129a679b7000e731b0477b286612"
server_key_iv: "f13ad041f7882feed1764d7565a41f7e"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.538574 21550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:54.550235 21550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:54.550958 21550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:54.551167 21550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:54.551779 21550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:54.552749 21550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:54.552824 21550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.552889 21550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:54.552938 21550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.562088 21550 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:42445
I20260924 10:38:54.562112 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:42445 every 8 connection(s)
I20260924 10:38:54.563061 21550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:54.571028 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21550
I20260924 10:38:54.571138 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:38:54.571424 2822 external_mini_cluster.cc:1428] Setting key fc1738b04db15a2acd1b9a6d51024c38
I20260924 10:38:54.572481 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.564873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:55389 (local address 127.2.193.190:46501)
0924 10:38:54.565010 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.565019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.566147 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 10:38:54.567659 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.567662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.567710 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.567787 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:54.568684 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.569332 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.570430 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.570653 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.570764 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.571931 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:54.571937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:54.571939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:54.571957 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:54.571984 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:54.571997 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:54.572068 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.572275 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.572340 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:38:54.573534 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.565310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46501 (local address 127.2.193.129:55389)
0924 10:38:54.566007 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.566049 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.567437 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.567957 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.567964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.568061 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:54.568548 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.568561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.569441 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.569444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.570326 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.570335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.570786 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.570789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.570927 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.571659 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:54.571661 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:54.571702 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:54.571768 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:54.572100 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:54.572109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:54.572115 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.572256 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.572510 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":162,"threads_started":1}
I20260924 10:38:54.573787 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46501
--builtin_ntp_servers=127.2.193.148:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:38:54.574880 21678 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46501
I20260924 10:38:54.575122 21678 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:54.575668 21678 heartbeater.cc:511] Master 127.2.193.190:46501 requested a full tablet report, sending...
I20260924 10:38:54.577345 21490 ts_manager.cc:194] Registered new tserver with Master: b93268d0e4874fafbd63bb8b625c81e4 (127.2.193.129:42445)
I20260924 10:38:54.579169 21490 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.129:55389
W20260924 10:38:54.687664 21681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:54.687951 21681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:54.687991 21681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:54.692090 21681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:54.692169 21681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:54.692297 21681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:54.697538 21681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:54.698665 21681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:54.700084 21681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:54.709764 21689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:54.709771 21687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:54.709805 21686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:54.709841 21681 server_base.cc:1034] running on GCE node
I20260924 10:38:54.710440 21681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:54.711149 21681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:54.712320 21681 hybrid_clock.cc:648] HybridClock initialized: now 1790246334712266 us; error 65 us; skew 500 ppm
I20260924 10:38:54.714244 21681 webserver.cc:460] Webserver started at http://127.2.193.130:35751/ using document root <none> and password file <none>
I20260924 10:38:54.714840 21681 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:54.714905 21681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:54.715124 21681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:54.717301 21681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "6db6321f61bc4d798099bb3f548799c9"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "1f59e1bdb829476d3b5fadccb5f31aa3"
server_key_iv: "d175793f3f596d12e7e0f04d3c9a9ad2"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.717890 21681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "6db6321f61bc4d798099bb3f548799c9"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "1f59e1bdb829476d3b5fadccb5f31aa3"
server_key_iv: "d175793f3f596d12e7e0f04d3c9a9ad2"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.721819 21681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:38:54.724385 21695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.725824 21681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 10:38:54.725956 21681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "6db6321f61bc4d798099bb3f548799c9"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "1f59e1bdb829476d3b5fadccb5f31aa3"
server_key_iv: "d175793f3f596d12e7e0f04d3c9a9ad2"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.726075 21681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:54.744719 21681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:54.745515 21681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:54.745724 21681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:54.746325 21681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:54.747270 21681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:54.747316 21681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.747390 21681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:54.747447 21681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.756217 21681 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:34191
I20260924 10:38:54.756292 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:34191 every 8 connection(s)
I20260924 10:38:54.757222 21681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:54.760785 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21681
I20260924 10:38:54.760936 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:38:54.761296 2822 external_mini_cluster.cc:1428] Setting key 3573cb9792036d47117587e69fd93089
I20260924 10:38:54.763218 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46501
--builtin_ntp_servers=127.2.193.148:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:38:54.766765 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.759082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:35795 (local address 127.2.193.190:46501)
0924 10:38:54.759222 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.759225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.760246 (+ 1021us) server_negotiation.cc:407] Connection header received
0924 10:38:54.761856 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.761859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.761922 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.762055 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:54.762989 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.763600 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.764776 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.765002 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.765116 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.766271 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:54.766278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:54.766281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:54.766293 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:54.766311 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:54.766319 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:54.766399 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.766564 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.766625 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:38:54.767831 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.759430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46501 (local address 127.2.193.130:35795)
0924 10:38:54.760088 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.760130 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.761633 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.762240 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.762248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.762341 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:54.762843 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.762856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.763731 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.763734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.764642 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.764651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.765159 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.765163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.765356 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.766003 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:54.766004 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:54.766046 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:54.766114 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:54.766403 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:54.766413 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:54.766423 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.766564 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.767038 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260924 10:38:54.769071 21809 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46501
I20260924 10:38:54.769387 21809 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:54.770102 21809 heartbeater.cc:511] Master 127.2.193.190:46501 requested a full tablet report, sending...
I20260924 10:38:54.771320 21490 ts_manager.cc:194] Registered new tserver with Master: 6db6321f61bc4d798099bb3f548799c9 (127.2.193.130:34191)
I20260924 10:38:54.771982 21490 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.130:35795
W20260924 10:38:54.880142 21812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:54.880421 21812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:54.880461 21812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:54.884678 21812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:54.884742 21812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:54.884835 21812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:54.889873 21812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:46501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:54.890990 21812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:54.892396 21812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:54.902374 21820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:54.902396 21818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:54.902383 21817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:54.902644 21812 server_base.cc:1034] running on GCE node
I20260924 10:38:54.903185 21812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:54.903860 21812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:54.905052 21812 hybrid_clock.cc:648] HybridClock initialized: now 1790246334905027 us; error 38 us; skew 500 ppm
I20260924 10:38:54.907006 21812 webserver.cc:460] Webserver started at http://127.2.193.131:41179/ using document root <none> and password file <none>
I20260924 10:38:54.907627 21812 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:54.907732 21812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:54.907969 21812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:54.910178 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "22a561d2ba564fb0bb5b76ed8dec38f4"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "b738c55ff3cf5f4c620235ddbc0d8925"
server_key_iv: "0b65abf9bab229dce3cd99a160f19dbf"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.910768 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "22a561d2ba564fb0bb5b76ed8dec38f4"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "b738c55ff3cf5f4c620235ddbc0d8925"
server_key_iv: "0b65abf9bab229dce3cd99a160f19dbf"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.914485 21812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:38:54.917002 21826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.918280 21812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:38:54.918409 21812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "22a561d2ba564fb0bb5b76ed8dec38f4"
format_stamp: "Formatted at 2026-09-24 10:38:54 on dist-test-slave-8wc7"
server_key: "b738c55ff3cf5f4c620235ddbc0d8925"
server_key_iv: "0b65abf9bab229dce3cd99a160f19dbf"
server_key_version: "encryptionkey@0"
I20260924 10:38:54.918538 21812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:54.928902 21812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:54.929580 21812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:54.929780 21812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:54.930353 21812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:54.931224 21812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:54.931301 21812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.931376 21812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:54.931411 21812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:54.939920 21812 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:34831
I20260924 10:38:54.939952 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:34831 every 8 connection(s)
I20260924 10:38:54.940841 21812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:54.949553 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.942672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:54411 (local address 127.2.193.190:46501)
0924 10:38:54.942817 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.942823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.943962 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 10:38:54.945322 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.945325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.945371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.945435 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:54.946268 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.946794 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.947842 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.948052 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.948127 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.949157 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:54.949159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:54.949161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:54.949188 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:54.949204 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:54.949211 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:54.949260 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.949381 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.949425 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:38:54.949954 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21812
I20260924 10:38:54.950028 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:38:54.950297 2822 external_mini_cluster.cc:1428] Setting key 9d12ef75d9e5756648281ff79627a30f
I20260924 10:38:54.950443 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.943139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:46501 (local address 127.2.193.131:54411)
0924 10:38:54.943812 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.943851 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.945074 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.945586 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.945593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.945674 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:54.946146 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.946159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.946896 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.946898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.947741 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.947750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.948176 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.948178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.948306 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.948931 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:54.948932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:54.948970 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:54.949027 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:54.949270 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:54.949275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:54.949279 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.949363 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.949564 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260924 10:38:54.951507 21940 heartbeater.cc:347] Connected to a master server at 127.2.193.190:46501
I20260924 10:38:54.951727 21940 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:54.952292 21940 heartbeater.cc:511] Master 127.2.193.190:46501 requested a full tablet report, sending...
I20260924 10:38:54.953322 21489 ts_manager.cc:194] Registered new tserver with Master: 22a561d2ba564fb0bb5b76ed8dec38f4 (127.2.193.131:34831)
I20260924 10:38:54.953859 21489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.131:54411
I20260924 10:38:54.964483 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:38:54.970638 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.966560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.2.193.190:46501)
0924 10:38:54.966728 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.966733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.966869 (+ 136us) server_negotiation.cc:407] Connection header received
0924 10:38:54.967030 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.967033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.967082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.967165 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:54.967520 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.968046 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.968918 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.969148 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.969279 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.970102 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:54.970106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:54.970108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:54.970126 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:54.970143 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:54.970152 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:54.970283 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.970430 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.970470 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:38:54.973755 21489 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:38:54.976697 21489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:38:54.991771 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.985562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34191 (local address 127.0.0.1:36634)
0924 10:38:54.986810 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.986824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.986915 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.987641 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.987643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.987656 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:54.987713 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.987717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.988761 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.988767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.990309 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.990316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.991377 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.991379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.991456 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.991485 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.991566 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.991653 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"mutex_wait_us":31,"thread_start_us":118,"threads_started":1}
I20260924 10:38:54.991840 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.986679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.2.193.130:34191)
0924 10:38:54.987062 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.987095 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.987119 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:38:54.987187 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.987192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.987361 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.987518 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:54.987859 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.988640 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.990438 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.991260 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.991374 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.991569 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.991622 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.991714 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":88,"threads_started":1}
I20260924 10:38:54.992385 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.986272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:34831 (local address 127.0.0.1:53678)
0924 10:38:54.986959 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.986972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.987059 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.987797 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.987803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.987821 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:54.987910 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.987917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.989307 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.989310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.990318 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.990328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.991976 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.991981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.992071 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.992110 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.992181 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.992257 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":73,"thread_start_us":88,"threads_started":1}
I20260924 10:38:54.992638 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.986430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.2.193.131:34831)
0924 10:38:54.987326 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.987336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.987355 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:38:54.987466 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.987471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.987647 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.987796 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:54.988027 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.989159 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.990483 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.991260 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.991374 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.991569 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.992230 (+ 661us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.992334 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":125,"threads_started":1}
I20260924 10:38:54.994098 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.985515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:42445 (local address 127.0.0.1:43164)
0924 10:38:54.986318 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:54.986369 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:54.986504 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:54.987159 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:54.987164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:54.987188 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:54.987328 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.987334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.988801 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.988804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.990305 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:54.990316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.993699 (+ 3383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.993702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:54.993775 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.993793 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:38:54.993890 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 10:38:54.993971 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":283,"thread_start_us":90,"threads_started":1}
I20260924 10:38:54.994180 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:54.985654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.2.193.129:42445)
0924 10:38:54.986048 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:54.986072 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:54.986652 (+ 580us) server_negotiation.cc:407] Connection header received
0924 10:38:54.986741 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:54.986747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:54.986990 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:54.987156 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:54.987476 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.988679 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.992045 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:54.993366 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:54.993482 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:54.993645 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:54.993922 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:38:54.994022 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":148,"threads_started":1}
I20260924 10:38:54.995314 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 07e80e669fcb4ad3ba5ea84748bc99b8 (DEFAULT_TABLE table=test-table [id=bc6b48d633724dcc80ca06db88398fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:54.995313 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 07e80e669fcb4ad3ba5ea84748bc99b8 (DEFAULT_TABLE table=test-table [id=bc6b48d633724dcc80ca06db88398fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:54.996488 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e80e669fcb4ad3ba5ea84748bc99b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:54.996488 21874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e80e669fcb4ad3ba5ea84748bc99b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:54.996701 21612 tablet_service.cc:1467] Processing CreateTablet for tablet 07e80e669fcb4ad3ba5ea84748bc99b8 (DEFAULT_TABLE table=test-table [id=bc6b48d633724dcc80ca06db88398fc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:54.997736 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e80e669fcb4ad3ba5ea84748bc99b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:55.003952 21957 tablet_bootstrap.cc:492] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: Bootstrap starting.
I20260924 10:38:55.005157 21959 tablet_bootstrap.cc:492] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Bootstrap starting.
I20260924 10:38:55.006361 21957 tablet_bootstrap.cc:654] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:55.006979 21959 tablet_bootstrap.cc:654] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:55.007205 21957 log.cc:824] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:55.007840 21959 log.cc:824] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:55.008391 21958 tablet_bootstrap.cc:492] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Bootstrap starting.
I20260924 10:38:55.009552 21957 tablet_bootstrap.cc:492] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: No bootstrap required, opened a new log
I20260924 10:38:55.009752 21957 ts_tablet_manager.cc:1397] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:38:55.010416 21958 tablet_bootstrap.cc:654] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:55.010533 21959 tablet_bootstrap.cc:492] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: No bootstrap required, opened a new log
I20260924 10:38:55.010727 21959 ts_tablet_manager.cc:1397] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 10:38:55.011162 21958 log.cc:824] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:55.012768 21958 tablet_bootstrap.cc:492] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: No bootstrap required, opened a new log
I20260924 10:38:55.012964 21958 ts_tablet_manager.cc:1397] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 10:38:55.014240 21957 raft_consensus.cc:348] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.014525 21957 raft_consensus.cc:374] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:55.014577 21957 raft_consensus.cc:729] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93268d0e4874fafbd63bb8b625c81e4, State: Initialized, Role: FOLLOWER
I20260924 10:38:55.014925 21959 raft_consensus.cc:348] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.015166 21959 raft_consensus.cc:374] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:55.015094 21957 consensus_queue.cc:260] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.015209 21959 raft_consensus.cc:729] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db6321f61bc4d798099bb3f548799c9, State: Initialized, Role: FOLLOWER
I20260924 10:38:55.015709 21959 consensus_queue.cc:260] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.016139 21957 ts_tablet_manager.cc:1428] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:38:55.016448 21809 heartbeater.cc:503] Master 127.2.193.190:46501 was elected leader, sending a full tablet report...
I20260924 10:38:55.016454 21678 heartbeater.cc:503] Master 127.2.193.190:46501 was elected leader, sending a full tablet report...
I20260924 10:38:55.016700 21959 ts_tablet_manager.cc:1428] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:38:55.017613 21958 raft_consensus.cc:348] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.017870 21958 raft_consensus.cc:374] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:55.017926 21958 raft_consensus.cc:729] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a561d2ba564fb0bb5b76ed8dec38f4, State: Initialized, Role: FOLLOWER
I20260924 10:38:55.018309 21958 consensus_queue.cc:260] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.018988 21940 heartbeater.cc:503] Master 127.2.193.190:46501 was elected leader, sending a full tablet report...
I20260924 10:38:55.019241 21958 ts_tablet_manager.cc:1428] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 10:38:55.026853 21964 raft_consensus.cc:482] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:55.027033 21964 raft_consensus.cc:504] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.028100 21964 leader_election.cc:288] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b93268d0e4874fafbd63bb8b625c81e4 (127.2.193.129:42445), 22a561d2ba564fb0bb5b76ed8dec38f4 (127.2.193.131:34831)
I20260924 10:38:55.031900 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:55.028321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:42445 (local address 127.2.193.130:43865)
0924 10:38:55.028589 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:55.028606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:55.028705 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:55.028971 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:55.028974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:55.028990 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:55.029108 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:55.029114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.029932 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.029935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:55.030814 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:55.030823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.031470 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.031473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:55.031580 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:55.031602 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:38:55.031695 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:38:55.031766 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260924 10:38:55.031900 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:55.028513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:43865 (local address 127.2.193.129:42445)
0924 10:38:55.028634 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:55.028640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:55.028668 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:38:55.028819 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:55.028822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:55.028873 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:55.028948 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:55.029262 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.029825 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.030956 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.031349 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.031478 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:55.031530 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:55.031720 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:38:55.031792 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:38:55.032173 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:55.028431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:45727 (local address 127.2.193.131:34831)
0924 10:38:55.028563 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:55.028570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:55.028895 (+ 325us) server_negotiation.cc:407] Connection header received
0924 10:38:55.029017 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:55.029020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:55.029063 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:55.029146 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:55.029455 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.030044 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.031199 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.031643 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.031767 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:55.031813 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:55.031999 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 10:38:55.032065 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 10:38:55.032189 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:55.028321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:34831 (local address 127.2.193.130:45727)
0924 10:38:55.028801 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:55.028812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:55.028931 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:55.029156 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:55.029159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:55.029214 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:55.029348 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:55.029353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.030145 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.030152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:55.031086 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:55.031095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.031768 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.031771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:55.031865 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:55.031890 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:38:55.031971 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:38:55.032038 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":108,"thread_start_us":137,"threads_started":1}
I20260924 10:38:55.032588 21632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e80e669fcb4ad3ba5ea84748bc99b8" candidate_uuid: "6db6321f61bc4d798099bb3f548799c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b93268d0e4874fafbd63bb8b625c81e4" is_pre_election: true
I20260924 10:38:55.032886 21632 raft_consensus.cc:2454] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6db6321f61bc4d798099bb3f548799c9 in term 0.
I20260924 10:38:55.032928 21894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e80e669fcb4ad3ba5ea84748bc99b8" candidate_uuid: "6db6321f61bc4d798099bb3f548799c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" is_pre_election: true
I20260924 10:38:55.033257 21894 raft_consensus.cc:2454] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6db6321f61bc4d798099bb3f548799c9 in term 0.
I20260924 10:38:55.033422 21697 leader_election.cc:302] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db6321f61bc4d798099bb3f548799c9, b93268d0e4874fafbd63bb8b625c81e4; no voters:
I20260924 10:38:55.033764 21964 raft_consensus.cc:2781] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:38:55.033864 21964 raft_consensus.cc:482] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:38:55.033926 21964 raft_consensus.cc:3037] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:55.035048 21964 raft_consensus.cc:504] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.035413 21964 leader_election.cc:288] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [CANDIDATE]: Term 1 election: Requested vote from peers b93268d0e4874fafbd63bb8b625c81e4 (127.2.193.129:42445), 22a561d2ba564fb0bb5b76ed8dec38f4 (127.2.193.131:34831)
I20260924 10:38:55.035688 21632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e80e669fcb4ad3ba5ea84748bc99b8" candidate_uuid: "6db6321f61bc4d798099bb3f548799c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b93268d0e4874fafbd63bb8b625c81e4"
I20260924 10:38:55.035740 21894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e80e669fcb4ad3ba5ea84748bc99b8" candidate_uuid: "6db6321f61bc4d798099bb3f548799c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4"
I20260924 10:38:55.035795 21632 raft_consensus.cc:3037] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:55.035882 21894 raft_consensus.cc:3037] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:55.036871 21894 raft_consensus.cc:2454] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6db6321f61bc4d798099bb3f548799c9 in term 1.
I20260924 10:38:55.036968 21632 raft_consensus.cc:2454] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6db6321f61bc4d798099bb3f548799c9 in term 1.
I20260924 10:38:55.037283 21698 leader_election.cc:302] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a561d2ba564fb0bb5b76ed8dec38f4, 6db6321f61bc4d798099bb3f548799c9; no voters:
I20260924 10:38:55.037453 21964 raft_consensus.cc:2781] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:55.037678 21964 raft_consensus.cc:686] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6db6321f61bc4d798099bb3f548799c9, State: Running, Role: LEADER
I20260924 10:38:55.038003 21964 consensus_queue.cc:237] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } }
I20260924 10:38:55.041159 21490 catalog_manager.cc:5495] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6db6321f61bc4d798099bb3f548799c9 (127.2.193.130). New cstate: current_term: 1 leader_uuid: "6db6321f61bc4d798099bb3f548799c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6db6321f61bc4d798099bb3f548799c9" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 } health_report { overall_health: UNKNOWN } } }
I20260924 10:38:55.055575 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:55.052319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.2.193.130:34191)
0924 10:38:55.052444 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:55.052449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:55.052469 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:38:55.052558 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:55.052560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:55.052615 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:55.052699 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:55.052891 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.053483 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.054283 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:55.054488 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:55.054575 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:55.054826 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:55.055230 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:38:55.055337 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:55.055407 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 10:38:55.055440 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:38:55.061586 21894 raft_consensus.cc:1261] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 1 FOLLOWER]: Refusing update from remote peer 6db6321f61bc4d798099bb3f548799c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:55.061586 21632 raft_consensus.cc:1261] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 1 FOLLOWER]: Refusing update from remote peer 6db6321f61bc4d798099bb3f548799c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:55.062294 21967 consensus_queue.cc:1035] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b93268d0e4874fafbd63bb8b625c81e4" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:55.062489 21964 consensus_queue.cc:1035] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a561d2ba564fb0bb5b76ed8dec38f4" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:38:55.064167 21679 tablet.cc:2369] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:55.070871 21972 mvcc.cc:205] Tried to move back new op lower bound from 7332848988404314112 to 7332848988318248960. Current Snapshot: MvccSnapshot[applied={T|T < 7332848988404314112}]
I20260924 10:38:55.080070 21489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:55.080269 21489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:55.082821 21489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e74eec9cb1e4ad799125dc6b4a46db7: Sending DeleteTablet for 3 replicas of tablet 07e80e669fcb4ad3ba5ea84748bc99b8
I20260924 10:38:55.083608 21612 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e80e669fcb4ad3ba5ea84748bc99b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:55 UTC) from {username='slave'} at 127.0.0.1:43164
I20260924 10:38:55.083802 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e80e669fcb4ad3ba5ea84748bc99b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:55 UTC) from {username='slave'} at 127.0.0.1:53678
I20260924 10:38:55.084067 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e80e669fcb4ad3ba5ea84748bc99b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:55 UTC) from {username='slave'} at 127.0.0.1:36634
I20260924 10:38:55.084321 21981 tablet_replica.cc:330] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4: stopping tablet replica
I20260924 10:38:55.084546 21982 tablet_replica.cc:330] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: stopping tablet replica
I20260924 10:38:55.084666 21981 raft_consensus.cc:2229] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:55.084985 21981 raft_consensus.cc:2258] T 07e80e669fcb4ad3ba5ea84748bc99b8 P b93268d0e4874fafbd63bb8b625c81e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:55.085084 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21550
I20260924 10:38:55.085124 21982 raft_consensus.cc:2229] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:38:55.086732 21982 raft_consensus.cc:2258] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:55.088406 21982 ts_tablet_manager.cc:1905] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:55.088579 21980 tablet_replica.cc:330] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: stopping tablet replica
I20260924 10:38:55.088865 21980 raft_consensus.cc:2229] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:55.089226 21980 raft_consensus.cc:2258] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:55.090840 21980 ts_tablet_manager.cc:1905] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:55.091727 21982 ts_tablet_manager.cc:1918] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:55.091962 21982 log.cc:1196] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/07e80e669fcb4ad3ba5ea84748bc99b8
I20260924 10:38:55.092371 21982 ts_tablet_manager.cc:1939] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 6db6321f61bc4d798099bb3f548799c9: Deleting consensus metadata
W20260924 10:38:55.092571 21475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:42445 (error 108)
I20260924 10:38:55.093389 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21681
I20260924 10:38:55.094147 21980 ts_tablet_manager.cc:1918] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:55.094234 21980 log.cc:1196] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/07e80e669fcb4ad3ba5ea84748bc99b8
I20260924 10:38:55.094524 21980 ts_tablet_manager.cc:1939] T 07e80e669fcb4ad3ba5ea84748bc99b8 P 22a561d2ba564fb0bb5b76ed8dec38f4: Deleting consensus metadata
I20260924 10:38:55.095794 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:55.095430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:42445 (local address 127.0.0.1:43170)
0924 10:38:55.095605 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:55.095690 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:42445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:38:55.095819 21476 catalog_manager.cc:4841] TS 22a561d2ba564fb0bb5b76ed8dec38f4 (127.2.193.131:34831): tablet 07e80e669fcb4ad3ba5ea84748bc99b8 (table test-table [id=bc6b48d633724dcc80ca06db88398fc8]) successfully deleted
W20260924 10:38:55.096076 21475 catalog_manager.cc:4584] TS b93268d0e4874fafbd63bb8b625c81e4 (127.2.193.129:42445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07e80e669fcb4ad3ba5ea84748bc99b8: Network error: Client connection negotiation failed: client connection to 127.2.193.129:42445: connect: Connection refused (error 111)
I20260924 10:38:55.099800 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21812
I20260924 10:38:55.102243 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:55.101944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34191 (local address 127.0.0.1:36656)
0924 10:38:55.102105 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:55.102163 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.130:34191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 10:38:55.105576 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21459
2026-09-24T10:38:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (994 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:38:55 dist-test-slave-8wc7 krb5kdc[21986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:38:55 dist-test-slave-8wc7 krb5kdc[21986](info): set up 2 sockets
Sep 24 10:38:55 dist-test-slave-8wc7 krb5kdc[21986](info): commencing operation
krb5kdc: starting...
W20260924 10:38:56.145354 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:38:56 dist-test-slave-8wc7 krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:38:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:38:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:38:56.275944 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:37829
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:37829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:56.387315 21999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:56.387607 21999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:56.387694 21999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:56.391834 21999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:56.391893 21999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:56.391912 21999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:56.391929 21999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:56.391971 21999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:56.397271 21999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:37829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:56.398346 21999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:56.399426 21999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:56.408022 22004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:56.408041 22007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:56.408077 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:56.408432 21999 server_base.cc:1034] running on GCE node
I20260924 10:38:56.409008 21999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:56.410207 21999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:56.411396 21999 hybrid_clock.cc:648] HybridClock initialized: now 1790246336411364 us; error 42 us; skew 500 ppm
Sep 24 10:38:56 dist-test-slave-8wc7 krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:56.414353 21999 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:38:56.415297 21999 webserver.cc:460] Webserver started at http://127.2.193.190:41987/ using document root <none> and password file <none>
I20260924 10:38:56.415867 21999 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:56.415961 21999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:56.416173 21999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:56.418292 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "24d1e1ee4e784cfcacff4d21fac75e3b"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "be5cc49df987afeca5ed4e935ee19bcb"
server_key_iv: "0d05fea6ba81de3b73248ad604de8e3a"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.418864 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "24d1e1ee4e784cfcacff4d21fac75e3b"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "be5cc49df987afeca5ed4e935ee19bcb"
server_key_iv: "0d05fea6ba81de3b73248ad604de8e3a"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.422638 21999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:38:56.425108 22014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:56.426470 21999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:38:56.426651 21999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "24d1e1ee4e784cfcacff4d21fac75e3b"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "be5cc49df987afeca5ed4e935ee19bcb"
server_key_iv: "0d05fea6ba81de3b73248ad604de8e3a"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.426775 21999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:56.443764 21999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:56.446808 21999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:56.447032 21999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:56.454299 21999 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:37829
I20260924 10:38:56.454320 22066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:37829 every 8 connection(s)
I20260924 10:38:56.455283 21999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:56.458362 22067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:56.462709 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 21999
I20260924 10:38:56.462869 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:38:56.463167 2822 external_mini_cluster.cc:1428] Setting key 9476eeb7d3ad85c68fc764b974cbb1e1
I20260924 10:38:56.464396 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b: Bootstrap starting.
I20260924 10:38:56.466833 22067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:56.467747 22067 log.cc:824] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:56.469976 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b: No bootstrap required, opened a new log
Sep 24 10:38:56 dist-test-slave-8wc7 krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:56.475114 22067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37829 } }
I20260924 10:38:56.475430 22067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:56.475517 22067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d1e1ee4e784cfcacff4d21fac75e3b, State: Initialized, Role: FOLLOWER
I20260924 10:38:56.476033 22067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37829 } }
I20260924 10:38:56.476204 22067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:56.476275 22067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:56.476377 22067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:56.477403 22067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37829 } }
I20260924 10:38:56.477805 22067 leader_election.cc:302] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24d1e1ee4e784cfcacff4d21fac75e3b; no voters:
I20260924 10:38:56.478148 22067 leader_election.cc:288] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:38:56.478257 22072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:56.478468 22072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [term 1 LEADER]: Becoming Leader. State: Replica: 24d1e1ee4e784cfcacff4d21fac75e3b, State: Running, Role: LEADER
I20260924 10:38:56.478781 22072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37829 } }
I20260924 10:38:56.478842 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:56.464660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.2.193.190:37829)
0924 10:38:56.465059 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:56.465082 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:56.465118 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:38:56.465822 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:56.465840 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:56.466172 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:56.466458 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:56.467414 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.468496 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.469649 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.470165 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.470281 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:56.473236 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:56.473246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:56.473270 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:56.473313 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:56.475341 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:56.475874 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:56.475879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:56.475886 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:56.476036 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:56.476359 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:56.476362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:56.476364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:56.476879 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:56.477127 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:56.477448 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 10:38:56.477771 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260924 10:38:56.479509 22067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:56.480041 22072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24d1e1ee4e784cfcacff4d21fac75e3b. Latest consensus state: current_term: 1 leader_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37829 } } }
I20260924 10:38:56.480154 22072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:56.480446 22073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d1e1ee4e784cfcacff4d21fac75e3b" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37829 } } }
I20260924 10:38:56.480556 22073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:56.481383 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:56.484511 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:56.490348 22076 catalog_manager.cc:1347] Generated new cluster ID: e4c865a27b1041fa84592a75813f32ee
I20260924 10:38:56.490429 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:56.497671 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:38:56.498594 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:56.507048 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b: Generated new TSK 0
I20260924 10:38:56.507802 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.193.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
Entry for principal kudu/127.2.193.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab.
I20260924 10:38:56.539856 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37829
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:56.654449 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:56.654743 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:56.654824 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:56.659015 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:56.659085 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:56.659188 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:56.664645 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.129.keytab
--principal=kudu/127.2.193.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:56.665895 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:56.666967 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:56.676823 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:56.676862 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:56.676919 22093 server_base.cc:1034] running on GCE node
W20260924 10:38:56.676862 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:56.677552 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:56.678146 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:56.679303 22093 hybrid_clock.cc:648] HybridClock initialized: now 1790246336679284 us; error 33 us; skew 500 ppm
Sep 24 10:38:56 dist-test-slave-8wc7 krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:56.682232 22093 init.cc:377] Logged in from keytab as kudu/127.2.193.129@KRBTEST.COM (short username kudu)
I20260924 10:38:56.683166 22093 webserver.cc:460] Webserver started at http://127.2.193.129:35271/ using document root <none> and password file <none>
I20260924 10:38:56.683744 22093 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:56.683837 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:56.684072 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:56.686228 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "9b6189f9d3dd4838b95b2664308ded4f"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "26c8c2c652b6a8e0edadaf340cfb9ae3"
server_key_iv: "c945cf8e393924d63610c2d5fc700a25"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.686830 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "9b6189f9d3dd4838b95b2664308ded4f"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "26c8c2c652b6a8e0edadaf340cfb9ae3"
server_key_iv: "c945cf8e393924d63610c2d5fc700a25"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.690711 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:38:56.693161 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:56.694492 22093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:38:56.694644 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "9b6189f9d3dd4838b95b2664308ded4f"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "26c8c2c652b6a8e0edadaf340cfb9ae3"
server_key_iv: "c945cf8e393924d63610c2d5fc700a25"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.694774 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:56.714959 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:56.718091 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:56.718319 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:56.718950 22093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:56.719828 22093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:56.719909 22093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:56.719978 22093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:56.720031 22093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:56.729514 22093 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:35317
I20260924 10:38:56.729540 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:35317 every 8 connection(s)
I20260924 10:38:56.730599 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:56.736595 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 22093
I20260924 10:38:56.736753 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:38:56.737058 2822 external_mini_cluster.cc:1428] Setting key 0ce2e8ec789c82cac787851e26d1b0c9
Sep 24 10:38:56 dist-test-slave-8wc7 krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.129@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:56.746591 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:56.732542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:36469 (local address 127.2.193.190:37829)
0924 10:38:56.732706 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:56.732711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:56.733707 (+ 996us) server_negotiation.cc:407] Connection header received
0924 10:38:56.734960 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:56.734963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:56.735012 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:56.735105 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:56.736371 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.737021 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.738554 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.738823 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.738910 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:56.742305 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:56.742309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:56.742311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:56.742337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:56.743941 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:56.744702 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:56.744706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:56.744707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:56.744796 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:56.745274 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:56.745276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:56.745278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:56.745497 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:56.745583 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:56.746313 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 10:38:56.746443 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.2.193.130@KRBTEST.COM; defaulting to no policy
I20260924 10:38:56.747906 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:56.732989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37829 (local address 127.2.193.129:36469)
0924 10:38:56.733552 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:56.733601 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:56.734749 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:56.735280 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:56.735288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:56.735668 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:56.736194 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:56.736209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.737167 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.737195 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:56.738426 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:56.738437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.739005 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.739011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:56.739263 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:56.739924 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:56.739950 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:56.742108 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:56.744078 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:56.744089 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:56.744110 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:56.744545 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:56.744898 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:56.744904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:56.744909 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:56.745123 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:56.745615 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:56.745621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:56.746001 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 10:38:56.746245 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 10:38:56.746665 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":162,"threads_started":1}
I20260924 10:38:56.749279 22222 heartbeater.cc:347] Connected to a master server at 127.2.193.190:37829
I20260924 10:38:56.749548 22222 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:56.750136 22222 heartbeater.cc:511] Master 127.2.193.190:37829 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.130@KRBTEST.COM" created.
I20260924 10:38:56.751830 22031 ts_manager.cc:194] Registered new tserver with Master: 9b6189f9d3dd4838b95b2664308ded4f (127.2.193.129:35317)
I20260924 10:38:56.753017 22031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.129@KRBTEST.COM'} at 127.2.193.129:36469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
Entry for principal kudu/127.2.193.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab.
I20260924 10:38:56.768532 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37829
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:56.882714 22227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:56.883054 22227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:56.883131 22227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:56.887423 22227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:56.887545 22227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:56.887650 22227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:56.893312 22227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.130.keytab
--principal=kudu/127.2.193.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:56.894529 22227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:56.895663 22227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:56.905788 22232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:56.905833 22235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:56.905987 22227 server_base.cc:1034] running on GCE node
W20260924 10:38:56.906177 22233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:56.906620 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:56.907297 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:56.908486 22227 hybrid_clock.cc:648] HybridClock initialized: now 1790246336908471 us; error 29 us; skew 500 ppm
Sep 24 10:38:56 dist-test-slave-8wc7 krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:56.911348 22227 init.cc:377] Logged in from keytab as kudu/127.2.193.130@KRBTEST.COM (short username kudu)
I20260924 10:38:56.912302 22227 webserver.cc:460] Webserver started at http://127.2.193.130:43861/ using document root <none> and password file <none>
I20260924 10:38:56.912887 22227 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:56.912982 22227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:56.913281 22227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:56.915366 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "689ee045b04148eeab40dc7d63f0c092"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "25e833744f6c42e866a1f69a79e9931c"
server_key_iv: "f2774cabd8827c75f78cee4e7a11b22d"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.915961 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "689ee045b04148eeab40dc7d63f0c092"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "25e833744f6c42e866a1f69a79e9931c"
server_key_iv: "f2774cabd8827c75f78cee4e7a11b22d"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.919809 22227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 10:38:56.922493 22242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:56.923784 22227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:38:56.923926 22227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "689ee045b04148eeab40dc7d63f0c092"
format_stamp: "Formatted at 2026-09-24 10:38:56 on dist-test-slave-8wc7"
server_key: "25e833744f6c42e866a1f69a79e9931c"
server_key_iv: "f2774cabd8827c75f78cee4e7a11b22d"
server_key_version: "encryptionkey@0"
I20260924 10:38:56.924052 22227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:56.942901 22227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:56.946498 22227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:56.946722 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:56.947366 22227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:56.948239 22227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:56.948312 22227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:56.948382 22227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:56.948433 22227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:56.957706 22227 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:42291
I20260924 10:38:56.957753 22355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:42291 every 8 connection(s)
I20260924 10:38:56.958637 22227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:56.965613 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 22227
I20260924 10:38:56.965775 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:38:56.966116 2822 external_mini_cluster.cc:1428] Setting key 0fc2195e654668c24c8bdcb053c3b936
Sep 24 10:38:56 dist-test-slave-8wc7 krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.130@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:56.973968 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:56.960445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:47057 (local address 127.2.193.190:37829)
0924 10:38:56.960581 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:56.960586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:56.961485 (+ 899us) server_negotiation.cc:407] Connection header received
0924 10:38:56.962693 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:56.962697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:56.962745 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:56.962821 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:56.963958 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.964496 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.965943 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.966280 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.966390 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:56.969563 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:56.969569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:56.969575 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:56.969610 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:56.971477 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:56.972049 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:56.972054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:56.972057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:56.972174 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:56.972607 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:56.972611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:56.972613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:56.972875 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:56.973010 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:56.973539 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 10:38:56.973688 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:38:56.974502 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:56.960833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37829 (local address 127.2.193.130:47057)
0924 10:38:56.961331 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:56.961374 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:56.962488 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:56.962990 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:56.962996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:56.963344 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:56.963825 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:56.963838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.964603 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.964606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:56.965776 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:56.965791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:56.966427 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:56.966430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:56.966650 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:56.967396 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:56.967435 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:56.969367 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:56.971604 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:56.971609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:56.971620 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:56.971865 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:56.972309 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:56.972314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:56.972315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:56.972466 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:56.973005 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:56.973010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:56.973271 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 10:38:56.973413 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:38:56.973647 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":104,"threads_started":1}
I20260924 10:38:56.975697 22356 heartbeater.cc:347] Connected to a master server at 127.2.193.190:37829
I20260924 10:38:56.975930 22356 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:56.976393 22356 heartbeater.cc:511] Master 127.2.193.190:37829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.193.131@KRBTEST.COM; defaulting to no policy
I20260924 10:38:56.977481 22031 ts_manager.cc:194] Registered new tserver with Master: 689ee045b04148eeab40dc7d63f0c092 (127.2.193.130:42291)
I20260924 10:38:56.978083 22031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.130@KRBTEST.COM'} at 127.2.193.130:47057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
Entry for principal kudu/127.2.193.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab.
I20260924 10:38:56.994417 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37829
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:57.108546 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:57.108888 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:57.108963 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:57.113067 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:57.113181 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:57.113301 22361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:57.118707 22361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.131.keytab
--principal=kudu/127.2.193.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:37829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:57.119867 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:57.120968 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:57.131100 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:57.131120 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:57.131098 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:57.131219 22361 server_base.cc:1034] running on GCE node
I20260924 10:38:57.131891 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:57.132730 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:57.133908 22361 hybrid_clock.cc:648] HybridClock initialized: now 1790246337133862 us; error 66 us; skew 500 ppm
Sep 24 10:38:57 dist-test-slave-8wc7 krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246337, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:57.136690 22361 init.cc:377] Logged in from keytab as kudu/127.2.193.131@KRBTEST.COM (short username kudu)
I20260924 10:38:57.137714 22361 webserver.cc:460] Webserver started at http://127.2.193.131:34817/ using document root <none> and password file <none>
I20260924 10:38:57.138294 22361 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:57.138391 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:57.138607 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:57.140700 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "18cd2537db55485ab94bd5888d9ba788"
format_stamp: "Formatted at 2026-09-24 10:38:57 on dist-test-slave-8wc7"
server_key: "30905ca2fbe3384991a56595e3d69c3b"
server_key_iv: "6d5ee29de6208d28e28916c7d8f5cff2"
server_key_version: "encryptionkey@0"
I20260924 10:38:57.141367 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "18cd2537db55485ab94bd5888d9ba788"
format_stamp: "Formatted at 2026-09-24 10:38:57 on dist-test-slave-8wc7"
server_key: "30905ca2fbe3384991a56595e3d69c3b"
server_key_iv: "6d5ee29de6208d28e28916c7d8f5cff2"
server_key_version: "encryptionkey@0"
I20260924 10:38:57.145109 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:57.147734 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:57.148978 22361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:38:57.149121 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "18cd2537db55485ab94bd5888d9ba788"
format_stamp: "Formatted at 2026-09-24 10:38:57 on dist-test-slave-8wc7"
server_key: "30905ca2fbe3384991a56595e3d69c3b"
server_key_iv: "6d5ee29de6208d28e28916c7d8f5cff2"
server_key_version: "encryptionkey@0"
I20260924 10:38:57.149293 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:57.168311 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:57.171578 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:57.171801 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:57.172509 22361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:57.173447 22361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:57.173517 22361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:57.173597 22361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:57.173640 22361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:57.182823 22361 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:45039
I20260924 10:38:57.182889 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:45039 every 8 connection(s)
I20260924 10:38:57.183825 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:57.190948 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 22361
I20260924 10:38:57.191095 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:38:57.191386 2822 external_mini_cluster.cc:1428] Setting key 1aba7688d1c91263bb8f4fbfc9fcb611
Sep 24 10:38:57 dist-test-slave-8wc7 krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246337, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.131@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:38:57.198920 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.185747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:37945 (local address 127.2.193.190:37829)
0924 10:38:57.185916 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.185921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.186794 (+ 873us) server_negotiation.cc:407] Connection header received
0924 10:38:57.188077 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.188080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.188129 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.188204 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:57.189500 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.190034 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.191616 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.191835 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.191965 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.194933 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:57.194936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:57.194938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:57.194964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:57.196558 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:57.197109 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:57.197112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:57.197114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:57.197277 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:57.197682 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:57.197685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:57.197686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:57.197888 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:57.197978 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.198678 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.198792 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:38:57.199879 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.186183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37829 (local address 127.2.193.131:37945)
0924 10:38:57.186638 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:57.186681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:57.187865 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:57.188374 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:57.188381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:57.188770 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:57.189368 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.189384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.190171 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.190174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.191485 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.191498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.192012 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.192015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.192235 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.193006 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:38:57.193028 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:57.194743 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:57.196685 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:57.196689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:57.196704 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:57.196983 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:57.197380 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:38:57.197382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:38:57.197384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:38:57.197576 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:38:57.197994 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:57.198000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:57.198317 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 10:38:57.198591 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 10:38:57.198889 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260924 10:38:57.201102 22490 heartbeater.cc:347] Connected to a master server at 127.2.193.190:37829
I20260924 10:38:57.201399 22490 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:57.201932 22490 heartbeater.cc:511] Master 127.2.193.190:37829 requested a full tablet report, sending...
I20260924 10:38:57.202986 22031 ts_manager.cc:194] Registered new tserver with Master: 18cd2537db55485ab94bd5888d9ba788 (127.2.193.131:45039)
I20260924 10:38:57.203533 22031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.193.131@KRBTEST.COM'} at 127.2.193.131:37945
I20260924 10:38:57.205564 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:38:57.215509 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.207556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.2.193.190:37829)
0924 10:38:57.207692 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.207696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.207871 (+ 175us) server_negotiation.cc:407] Connection header received
0924 10:38:57.208032 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.208036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.208088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.208172 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:57.209126 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.209826 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.210720 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.210965 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.211078 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.212473 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:57.212477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:57.212480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:38:57.212506 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:57.213818 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:57.214242 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:57.214246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:57.214248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:57.214330 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:38:57.214607 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:38:57.214612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:38:57.214614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:38:57.214837 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:57.214990 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.215217 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.215339 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:38:57.218359 22031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:38:57.221473 22031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:38:57.237010 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.230361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.2.193.130:42291)
0924 10:38:57.231735 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.231748 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.231774 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:38:57.231851 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.231856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.232084 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.232262 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:57.232551 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.233814 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.235550 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.236325 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.236424 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.236538 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.236753 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.236853 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":107,"threads_started":1}
I20260924 10:38:57.237041 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.230260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:42291 (local address 127.0.0.1:48044)
0924 10:38:57.230788 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:57.230838 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:57.230974 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:57.232251 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:57.232257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:57.232278 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:57.232437 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.232444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.233955 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.233958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.235422 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.235431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.236454 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.236458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.236594 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.236610 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 10:38:57.236699 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:38:57.236868 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":144,"spinlock_wait_cycles":5248,"thread_start_us":88,"threads_started":1}
I20260924 10:38:57.237041 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.231066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:45039 (local address 127.0.0.1:51060)
0924 10:38:57.231317 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:57.231329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:57.231413 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:57.232427 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:57.232429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:57.232441 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:57.232539 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.232548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.233992 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.233996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.235189 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.235198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.236310 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.236313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.236456 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.236573 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 10:38:57.236699 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 10:38:57.236868 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":216,"thread_start_us":117,"threads_started":1}
I20260924 10:38:57.237870 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.231239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.2.193.131:45039)
0924 10:38:57.231777 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.231794 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.231821 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:38:57.231897 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.231902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.232111 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.232264 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:57.232645 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.233821 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.235320 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.236168 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.236290 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.236457 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.237246 (+ 789us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.237739 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":91,"threads_started":1}
I20260924 10:38:57.240015 22290 tablet_service.cc:1467] Processing CreateTablet for tablet 0012fd36f3724ca4a30fdaf8d8730997 (DEFAULT_TABLE table=test-table [id=ae0c16570c564a8690903ed9cc78aa68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:57.240139 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 0012fd36f3724ca4a30fdaf8d8730997 (DEFAULT_TABLE table=test-table [id=ae0c16570c564a8690903ed9cc78aa68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:57.241034 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0012fd36f3724ca4a30fdaf8d8730997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:57.241019 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0012fd36f3724ca4a30fdaf8d8730997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:57.246356 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.230399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35317 (local address 127.0.0.1:35810)
0924 10:38:57.230788 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:57.230838 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:57.230971 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:57.232360 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:57.232363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:57.232378 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:57.232514 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.232522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.233724 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.233729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.235420 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.235431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.245831 (+ 10400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.245835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.245967 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.246038 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 10:38:57.246099 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 10:38:57.246197 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":155,"spinlock_wait_cycles":3840,"thread_start_us":115,"threads_started":1}
I20260924 10:38:57.247184 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.231400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.2.193.129:35317)
0924 10:38:57.231967 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.231981 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.232000 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:38:57.232062 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.232066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.232220 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.232671 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:57.232802 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.233589 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.236605 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.245646 (+ 9041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.246691 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.246864 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.246924 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.247044 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":81,"threads_started":1}
I20260924 10:38:57.247577 22507 tablet_bootstrap.cc:492] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Bootstrap starting.
I20260924 10:38:57.248862 22508 tablet_bootstrap.cc:492] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Bootstrap starting.
I20260924 10:38:57.250500 22507 tablet_bootstrap.cc:654] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:57.250299 22156 tablet_service.cc:1467] Processing CreateTablet for tablet 0012fd36f3724ca4a30fdaf8d8730997 (DEFAULT_TABLE table=test-table [id=ae0c16570c564a8690903ed9cc78aa68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:57.250844 22508 tablet_bootstrap.cc:654] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:57.251406 22507 log.cc:824] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:57.251469 22156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0012fd36f3724ca4a30fdaf8d8730997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:57.251999 22508 log.cc:824] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:57.253719 22507 tablet_bootstrap.cc:492] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: No bootstrap required, opened a new log
I20260924 10:38:57.253918 22507 ts_tablet_manager.cc:1397] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:38:57.253966 22508 tablet_bootstrap.cc:492] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: No bootstrap required, opened a new log
I20260924 10:38:57.254153 22508 ts_tablet_manager.cc:1397] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:38:57.257351 22508 raft_consensus.cc:348] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.257633 22508 raft_consensus.cc:374] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:57.257688 22508 raft_consensus.cc:729] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cd2537db55485ab94bd5888d9ba788, State: Initialized, Role: FOLLOWER
I20260924 10:38:57.257880 22507 raft_consensus.cc:348] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.258154 22507 raft_consensus.cc:374] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:57.258209 22507 raft_consensus.cc:729] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689ee045b04148eeab40dc7d63f0c092, State: Initialized, Role: FOLLOWER
I20260924 10:38:57.258167 22508 consensus_queue.cc:260] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.258630 22507 consensus_queue.cc:260] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.258832 22490 heartbeater.cc:503] Master 127.2.193.190:37829 was elected leader, sending a full tablet report...
I20260924 10:38:57.259212 22508 ts_tablet_manager.cc:1428] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:38:57.259578 22507 ts_tablet_manager.cc:1428] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:38:57.259953 22356 heartbeater.cc:503] Master 127.2.193.190:37829 was elected leader, sending a full tablet report...
I20260924 10:38:57.260159 22512 tablet_bootstrap.cc:492] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: Bootstrap starting.
I20260924 10:38:57.262650 22512 tablet_bootstrap.cc:654] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:57.263530 22512 log.cc:824] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:57.265619 22512 tablet_bootstrap.cc:492] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: No bootstrap required, opened a new log
I20260924 10:38:57.265836 22512 ts_tablet_manager.cc:1397] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 10:38:57.269232 22512 raft_consensus.cc:348] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.269492 22512 raft_consensus.cc:374] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:57.269580 22512 raft_consensus.cc:729] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6189f9d3dd4838b95b2664308ded4f, State: Initialized, Role: FOLLOWER
I20260924 10:38:57.270041 22512 consensus_queue.cc:260] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.270758 22222 heartbeater.cc:503] Master 127.2.193.190:37829 was elected leader, sending a full tablet report...
I20260924 10:38:57.271070 22512 ts_tablet_manager.cc:1428] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 10:38:57.434923 22491 tablet.cc:2369] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:38:57.459837 22357 tablet.cc:2369] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:57.464905 22511 raft_consensus.cc:482] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:57.465154 22511 raft_consensus.cc:504] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.466416 22511 leader_election.cc:288] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 689ee045b04148eeab40dc7d63f0c092 (127.2.193.130:42291), 9b6189f9d3dd4838b95b2664308ded4f (127.2.193.129:35317)
I20260924 10:38:57.471230 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.466728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35317 (local address 127.2.193.131:40477)
0924 10:38:57.466879 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:57.466896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:57.467001 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:57.467351 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:57.467354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:57.467377 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:57.467567 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.467576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.468611 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.468614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.469888 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.469899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.470756 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.470758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.470885 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.470912 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:38:57.470992 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:38:57.471078 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":24}
I20260924 10:38:57.471249 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.466962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:40477 (local address 127.2.193.129:35317)
0924 10:38:57.467085 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.467089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.467104 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:38:57.467158 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.467161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.467234 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.467314 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:57.467710 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.468495 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.470019 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.470637 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.470762 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.470832 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.471021 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.471126 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 10:38:57.471256 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.466818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:45357 (local address 127.2.193.130:42291)
0924 10:38:57.466949 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.466953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.467578 (+ 625us) server_negotiation.cc:407] Connection header received
0924 10:38:57.467665 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.467672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.467727 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.467805 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:57.468145 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.468769 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.470175 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.470750 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.470859 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.470906 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.471092 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.471158 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:38:57.471238 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.466729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:42291 (local address 127.2.193.131:45357)
0924 10:38:57.467434 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:57.467448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:57.467577 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:57.467837 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:57.467840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:57.467855 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:57.467992 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.467999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.468875 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.468879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.470059 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:57.470067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.470858 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.470861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:57.470952 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.470993 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:38:57.471057 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:38:57.471104 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":274,"thread_start_us":175,"threads_started":1}
I20260924 10:38:57.472066 22176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0012fd36f3724ca4a30fdaf8d8730997" candidate_uuid: "18cd2537db55485ab94bd5888d9ba788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6189f9d3dd4838b95b2664308ded4f" is_pre_election: true
I20260924 10:38:57.472074 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0012fd36f3724ca4a30fdaf8d8730997" candidate_uuid: "18cd2537db55485ab94bd5888d9ba788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689ee045b04148eeab40dc7d63f0c092" is_pre_election: true
I20260924 10:38:57.472404 22176 raft_consensus.cc:2454] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cd2537db55485ab94bd5888d9ba788 in term 0.
I20260924 10:38:57.472404 22310 raft_consensus.cc:2454] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cd2537db55485ab94bd5888d9ba788 in term 0.
I20260924 10:38:57.472898 22379 leader_election.cc:302] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cd2537db55485ab94bd5888d9ba788, 9b6189f9d3dd4838b95b2664308ded4f; no voters:
I20260924 10:38:57.473152 22511 raft_consensus.cc:2781] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:38:57.473304 22511 raft_consensus.cc:482] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:38:57.473363 22511 raft_consensus.cc:3037] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:57.474418 22511 raft_consensus.cc:504] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.474817 22511 leader_election.cc:288] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [CANDIDATE]: Term 1 election: Requested vote from peers 689ee045b04148eeab40dc7d63f0c092 (127.2.193.130:42291), 9b6189f9d3dd4838b95b2664308ded4f (127.2.193.129:35317)
I20260924 10:38:57.475191 22176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0012fd36f3724ca4a30fdaf8d8730997" candidate_uuid: "18cd2537db55485ab94bd5888d9ba788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6189f9d3dd4838b95b2664308ded4f"
I20260924 10:38:57.475191 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0012fd36f3724ca4a30fdaf8d8730997" candidate_uuid: "18cd2537db55485ab94bd5888d9ba788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689ee045b04148eeab40dc7d63f0c092"
I20260924 10:38:57.475322 22310 raft_consensus.cc:3037] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:57.475322 22176 raft_consensus.cc:3037] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:57.476626 22310 raft_consensus.cc:2454] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cd2537db55485ab94bd5888d9ba788 in term 1.
I20260924 10:38:57.476627 22176 raft_consensus.cc:2454] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cd2537db55485ab94bd5888d9ba788 in term 1.
I20260924 10:38:57.476949 22379 leader_election.cc:302] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cd2537db55485ab94bd5888d9ba788, 9b6189f9d3dd4838b95b2664308ded4f; no voters:
I20260924 10:38:57.477130 22511 raft_consensus.cc:2781] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:57.477381 22511 raft_consensus.cc:686] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 1 LEADER]: Becoming Leader. State: Replica: 18cd2537db55485ab94bd5888d9ba788, State: Running, Role: LEADER
I20260924 10:38:57.477722 22511 consensus_queue.cc:237] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } }
I20260924 10:38:57.481277 22031 catalog_manager.cc:5495] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cd2537db55485ab94bd5888d9ba788 (127.2.193.131). New cstate: current_term: 1 leader_uuid: "18cd2537db55485ab94bd5888d9ba788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cd2537db55485ab94bd5888d9ba788" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 45039 } health_report { overall_health: HEALTHY } } }
W20260924 10:38:57.482000 22223 tablet.cc:2369] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:57.533305 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.529800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.2.193.131:45039)
0924 10:38:57.529924 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.529927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.529940 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:38:57.530068 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.530070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.530126 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.530217 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:57.530499 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.531004 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.531884 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.532098 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.532176 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.532453 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:57.532932 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:38:57.533032 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.533080 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.533131 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:38:57.539448 22176 raft_consensus.cc:1261] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 1 FOLLOWER]: Refusing update from remote peer 18cd2537db55485ab94bd5888d9ba788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:57.539451 22310 raft_consensus.cc:1261] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 1 FOLLOWER]: Refusing update from remote peer 18cd2537db55485ab94bd5888d9ba788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:57.540206 22517 consensus_queue.cc:1035] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "689ee045b04148eeab40dc7d63f0c092" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 42291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:57.540400 22511 consensus_queue.cc:1035] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b6189f9d3dd4838b95b2664308ded4f" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 35317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:57.556595 22030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:57.556792 22030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:57.559345 22030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24d1e1ee4e784cfcacff4d21fac75e3b: Sending DeleteTablet for 3 replicas of tablet 0012fd36f3724ca4a30fdaf8d8730997
I20260924 10:38:57.560130 22290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0012fd36f3724ca4a30fdaf8d8730997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:57 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:48044
I20260924 10:38:57.560204 22156 tablet_service.cc:1514] Processing DeleteTablet for tablet 0012fd36f3724ca4a30fdaf8d8730997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:57 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:35810
I20260924 10:38:57.560864 22531 tablet_replica.cc:330] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f: stopping tablet replica
I20260924 10:38:57.560810 22424 tablet_service.cc:1514] Processing DeleteTablet for tablet 0012fd36f3724ca4a30fdaf8d8730997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:57 UTC) from {username='kudu', principal='kudu/127.2.193.190@KRBTEST.COM'} at 127.0.0.1:51060
I20260924 10:38:57.560940 22530 tablet_replica.cc:330] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: stopping tablet replica
I20260924 10:38:57.561298 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 22093
I20260924 10:38:57.561288 22530 raft_consensus.cc:2229] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:57.561288 22531 raft_consensus.cc:2229] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:57.561631 22530 raft_consensus.cc:2258] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:57.561650 22531 raft_consensus.cc:2258] T 0012fd36f3724ca4a30fdaf8d8730997 P 9b6189f9d3dd4838b95b2664308ded4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:57.562968 22530 ts_tablet_manager.cc:1905] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:57.562126 22532 tablet_replica.cc:330] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: stopping tablet replica
I20260924 10:38:57.565342 22530 ts_tablet_manager.cc:1918] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:57.565454 22532 raft_consensus.cc:2229] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:38:57.565500 22530 log.cc:1196] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/0012fd36f3724ca4a30fdaf8d8730997
I20260924 10:38:57.565857 22532 raft_consensus.cc:2258] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:57.566006 22530 ts_tablet_manager.cc:1939] T 0012fd36f3724ca4a30fdaf8d8730997 P 689ee045b04148eeab40dc7d63f0c092: Deleting consensus metadata
I20260924 10:38:57.567382 22016 catalog_manager.cc:4841] TS 689ee045b04148eeab40dc7d63f0c092 (127.2.193.130:42291): tablet 0012fd36f3724ca4a30fdaf8d8730997 (table test-table [id=ae0c16570c564a8690903ed9cc78aa68]) successfully deleted
I20260924 10:38:57.567953 22532 ts_tablet_manager.cc:1905] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:38:57.569329 22017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:35317 (error 108)
I20260924 10:38:57.570616 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.570301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:35317 (local address 127.0.0.1:35824)
0924 10:38:57.570444 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:57.570524 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:35317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 10:38:57.570814 22532 ts_tablet_manager.cc:1918] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:57.570932 22532 log.cc:1196] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/0012fd36f3724ca4a30fdaf8d8730997
W20260924 10:38:57.571019 22017 catalog_manager.cc:4584] TS 9b6189f9d3dd4838b95b2664308ded4f (127.2.193.129:35317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0012fd36f3724ca4a30fdaf8d8730997: Network error: Client connection negotiation failed: client connection to 127.2.193.129:35317: connect: Connection refused (error 111)
I20260924 10:38:57.571287 22532 ts_tablet_manager.cc:1939] T 0012fd36f3724ca4a30fdaf8d8730997 P 18cd2537db55485ab94bd5888d9ba788: Deleting consensus metadata
I20260924 10:38:57.571345 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 22227
I20260924 10:38:57.572445 22017 catalog_manager.cc:4841] TS 18cd2537db55485ab94bd5888d9ba788 (127.2.193.131:45039): tablet 0012fd36f3724ca4a30fdaf8d8730997 (table test-table [id=ae0c16570c564a8690903ed9cc78aa68]) successfully deleted
I20260924 10:38:57.577906 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 22361
I20260924 10:38:57.583986 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 21999
2026-09-24T10:38:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2479 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T10:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:38:57Z Disabled control of system clock
I20260924 10:38:57.614563 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:33121
--webserver_interface=127.2.193.190
--webserver_port=0
--builtin_ntp_servers=127.2.193.148:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:33121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:38:57.726406 22537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:57.726717 22537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:57.726755 22537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:57.730921 22537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:57.730995 22537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:57.731011 22537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:57.731030 22537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:57.731046 22537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:57.735980 22537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:33121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:33121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:57.737106 22537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:57.738637 22537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:57.747761 22542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:57.747855 22543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:57.747864 22537 server_base.cc:1034] running on GCE node
W20260924 10:38:57.748100 22545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:57.748728 22537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:57.750030 22537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:57.751227 22537 hybrid_clock.cc:648] HybridClock initialized: now 1790246337751191 us; error 54 us; skew 500 ppm
I20260924 10:38:57.753507 22537 webserver.cc:460] Webserver started at http://127.2.193.190:46829/ using document root <none> and password file <none>
I20260924 10:38:57.754132 22537 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:57.754199 22537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:57.754427 22537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:57.756631 22537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "4e1bf121ced14b71828f283ece4a1fd7"
format_stamp: "Formatted at 2026-09-24 10:38:57 on dist-test-slave-8wc7"
server_key: "a9a2325f39606ed7e9b5689484e37fcd"
server_key_iv: "f044dcae5471272b2a7f5001686cb7cd"
server_key_version: "encryptionkey@0"
I20260924 10:38:57.757332 22537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "4e1bf121ced14b71828f283ece4a1fd7"
format_stamp: "Formatted at 2026-09-24 10:38:57 on dist-test-slave-8wc7"
server_key: "a9a2325f39606ed7e9b5689484e37fcd"
server_key_iv: "f044dcae5471272b2a7f5001686cb7cd"
server_key_version: "encryptionkey@0"
I20260924 10:38:57.761103 22537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:38:57.763741 22551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:57.765120 22537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 10:38:57.765316 22537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "4e1bf121ced14b71828f283ece4a1fd7"
format_stamp: "Formatted at 2026-09-24 10:38:57 on dist-test-slave-8wc7"
server_key: "a9a2325f39606ed7e9b5689484e37fcd"
server_key_iv: "f044dcae5471272b2a7f5001686cb7cd"
server_key_version: "encryptionkey@0"
I20260924 10:38:57.765451 22537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:57.781960 22537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:57.782708 22537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:57.782923 22537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:57.790299 22537 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:33121
I20260924 10:38:57.790330 22603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:33121 every 8 connection(s)
I20260924 10:38:57.791498 22537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:38:57.794699 22604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:57.800177 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 22537
I20260924 10:38:57.800290 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:38:57.800601 2822 external_mini_cluster.cc:1428] Setting key 83881875134a44fdc39f42beaec955e7
I20260924 10:38:57.800580 22604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7: Bootstrap starting.
I20260924 10:38:57.803125 22604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:57.803891 22604 log.cc:824] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:57.806051 22604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7: No bootstrap required, opened a new log
I20260924 10:38:57.810259 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:57.802069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.2.193.190:33121)
0924 10:38:57.802720 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:57.802739 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:57.802776 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:38:57.803489 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:57.803506 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:57.803917 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:57.804265 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:57.804748 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.805798 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.807001 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:57.807471 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:57.807590 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:57.808429 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:57.808434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:57.808447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:57.808829 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:57.808884 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:57.808904 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:57.808988 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:57.809167 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 10:38:57.809516 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":151,"threads_started":1}
I20260924 10:38:57.810230 22604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1bf121ced14b71828f283ece4a1fd7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 33121 } }
I20260924 10:38:57.810501 22604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:57.810549 22604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1bf121ced14b71828f283ece4a1fd7, State: Initialized, Role: FOLLOWER
I20260924 10:38:57.810930 22604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1bf121ced14b71828f283ece4a1fd7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 33121 } }
I20260924 10:38:57.811031 22604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:38:57.811071 22604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:38:57.811136 22604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:57.812237 22604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1bf121ced14b71828f283ece4a1fd7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 33121 } }
I20260924 10:38:57.812608 22604 leader_election.cc:302] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1bf121ced14b71828f283ece4a1fd7; no voters:
I20260924 10:38:57.812930 22604 leader_election.cc:288] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:38:57.813090 22609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:57.813390 22609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [term 1 LEADER]: Becoming Leader. State: Replica: 4e1bf121ced14b71828f283ece4a1fd7, State: Running, Role: LEADER
I20260924 10:38:57.813727 22609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1bf121ced14b71828f283ece4a1fd7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 33121 } }
I20260924 10:38:57.814217 22604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:38:57.815289 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e1bf121ced14b71828f283ece4a1fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1bf121ced14b71828f283ece4a1fd7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 33121 } } }
I20260924 10:38:57.815424 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:38:57.815716 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1bf121ced14b71828f283ece4a1fd7. Latest consensus state: current_term: 1 leader_uuid: "4e1bf121ced14b71828f283ece4a1fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1bf121ced14b71828f283ece4a1fd7" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 33121 } } }
I20260924 10:38:57.815852 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7 [sys.catalog]: This master's current role is: LEADER
W20260924 10:38:57.818838 22624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:38:57.818915 22624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:38:57.818998 22618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:38:57.819906 22618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:38:57.825811 22618 catalog_manager.cc:1347] Generated new cluster ID: f4cbb4ee3ae74921ba1c21a6da28fa8f
I20260924 10:38:57.825882 22618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:38:57.846957 22618 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:38:57.847971 22618 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:38:57.856843 22618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7: Generated new TSK 0
I20260924 10:38:57.857661 22618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:38:57.863183 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.129:0
--local_ip_for_outbound_sockets=127.2.193.129
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:33121
--builtin_ntp_servers=127.2.193.148:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:38:57.978433 22628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:57.978753 22628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:57.978803 22628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:57.982775 22628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:57.982823 22628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:57.982894 22628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.129
I20260924 10:38:57.987547 22628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.193.129
--webserver_port=0
--tserver_master_addrs=127.2.193.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.129
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:57.988608 22628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:57.990381 22628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:58.000061 22633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:58.000232 22628 server_base.cc:1034] running on GCE node
W20260924 10:38:58.000141 22634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:58.000129 22636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:58.000779 22628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:58.001506 22628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:58.002684 22628 hybrid_clock.cc:648] HybridClock initialized: now 1790246338002669 us; error 29 us; skew 500 ppm
I20260924 10:38:58.004506 22628 webserver.cc:460] Webserver started at http://127.2.193.129:41833/ using document root <none> and password file <none>
I20260924 10:38:58.005139 22628 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:58.005285 22628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:58.005501 22628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:58.007678 22628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data/instance:
uuid: "776b22c129f246f2b1686f939adbbbbb"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "25e1cd11454c6bee6cf884b0b5c8164e"
server_key_iv: "082c6272c160432412c8e450b5708a37"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.008267 22628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance:
uuid: "776b22c129f246f2b1686f939adbbbbb"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "25e1cd11454c6bee6cf884b0b5c8164e"
server_key_iv: "082c6272c160432412c8e450b5708a37"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.012077 22628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 10:38:58.014708 22642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.016080 22628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:38:58.016197 22628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/wal
uuid: "776b22c129f246f2b1686f939adbbbbb"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "25e1cd11454c6bee6cf884b0b5c8164e"
server_key_iv: "082c6272c160432412c8e450b5708a37"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.016311 22628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:58.040294 22628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:58.041044 22628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:58.041311 22628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:58.041942 22628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:58.042932 22628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:58.043006 22628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.043089 22628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:58.043133 22628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.052515 22628 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.129:40281
I20260924 10:38:58.052552 22755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.129:40281 every 8 connection(s)
I20260924 10:38:58.053967 22628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/data/info.pb
I20260924 10:38:58.060112 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 22628
I20260924 10:38:58.060252 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-0/wal/instance
I20260924 10:38:58.060593 2822 external_mini_cluster.cc:1428] Setting key 0fcbe73b6f6641c446d2ae9a9fe23c64
I20260924 10:38:58.063115 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.130:0
--local_ip_for_outbound_sockets=127.2.193.130
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:33121
--builtin_ntp_servers=127.2.193.148:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:38:58.065140 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.055946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:58513 (local address 127.2.193.190:33121)
0924 10:38:58.056093 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.056097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.057323 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 10:38:58.058778 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.058782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.058836 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.058927 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:58.059866 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.060566 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.062440 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.062828 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.062930 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.064612 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:58.064616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:58.064618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:58.064633 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:58.064658 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:58.064666 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:58.064743 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.064930 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.064996 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:38:58.066124 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.056363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:33121 (local address 127.2.193.129:58513)
0924 10:38:58.057120 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.057201 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.058528 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.059100 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.059107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.059196 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:58.059710 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.059723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.060763 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.060769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.062270 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.062283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.063194 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.063200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.063459 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.064310 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:58.064311 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:58.064358 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:58.064438 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:58.064758 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:58.064768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:58.064774 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.064927 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.065266 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260924 10:38:58.067627 22756 heartbeater.cc:347] Connected to a master server at 127.2.193.190:33121
I20260924 10:38:58.067847 22756 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:58.068361 22756 heartbeater.cc:511] Master 127.2.193.190:33121 requested a full tablet report, sending...
I20260924 10:38:58.070107 22568 ts_manager.cc:194] Registered new tserver with Master: 776b22c129f246f2b1686f939adbbbbb (127.2.193.129:40281)
I20260924 10:38:58.071334 22568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.129:58513
W20260924 10:38:58.179610 22759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:58.179934 22759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:58.180011 22759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:58.184538 22759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:58.184677 22759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:58.184782 22759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.130
I20260924 10:38:58.190235 22759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.193.130
--webserver_port=0
--tserver_master_addrs=127.2.193.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.130
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:58.191404 22759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:58.192862 22759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:38:58.202764 22759 server_base.cc:1034] running on GCE node
W20260924 10:38:58.202713 22767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:58.202798 22765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:58.202714 22764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:58.203464 22759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:58.204154 22759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:58.205344 22759 hybrid_clock.cc:648] HybridClock initialized: now 1790246338205327 us; error 29 us; skew 500 ppm
I20260924 10:38:58.207261 22759 webserver.cc:460] Webserver started at http://127.2.193.130:41627/ using document root <none> and password file <none>
I20260924 10:38:58.207902 22759 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:58.208009 22759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:58.208244 22759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:58.210405 22759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data/instance:
uuid: "f2ae1882d7ee42198ed66ec9486c884b"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "ceaa77241a903ba5b269d8b4f1207cb7"
server_key_iv: "7463929ea3923f69b4b3f98f3659fa60"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.210981 22759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance:
uuid: "f2ae1882d7ee42198ed66ec9486c884b"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "ceaa77241a903ba5b269d8b4f1207cb7"
server_key_iv: "7463929ea3923f69b4b3f98f3659fa60"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.214804 22759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 10:38:58.217515 22773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.218783 22759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:38:58.218922 22759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal
uuid: "f2ae1882d7ee42198ed66ec9486c884b"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "ceaa77241a903ba5b269d8b4f1207cb7"
server_key_iv: "7463929ea3923f69b4b3f98f3659fa60"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.219054 22759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:58.231426 22759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:58.232136 22759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:58.232343 22759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:58.232947 22759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:58.233940 22759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:58.234026 22759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.234098 22759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:58.234149 22759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.244221 22759 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.130:34573
I20260924 10:38:58.244527 22886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.130:34573 every 8 connection(s)
I20260924 10:38:58.245568 22759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/data/info.pb
I20260924 10:38:58.250468 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 22759
I20260924 10:38:58.250631 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal/instance
I20260924 10:38:58.250957 2822 external_mini_cluster.cc:1428] Setting key e4805d0e30ba118f9843f29edb0a569d
I20260924 10:38:58.253003 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.193.131:0
--local_ip_for_outbound_sockets=127.2.193.131
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:33121
--builtin_ntp_servers=127.2.193.148:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:38:58.256259 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.247788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.130:56229 (local address 127.2.193.190:33121)
0924 10:38:58.247975 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.247979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.249267 (+ 1288us) server_negotiation.cc:407] Connection header received
0924 10:38:58.251043 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.251049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.251114 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.251196 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:58.252339 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.252882 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.254196 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.254440 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.254566 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.255732 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:58.255736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:58.255739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:58.255753 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:58.255776 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:58.255790 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:58.255866 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.256050 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.256116 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 10:38:58.257258 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.248323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:33121 (local address 127.2.193.130:56229)
0924 10:38:58.249076 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.249118 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.250755 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.251433 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.251441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.251564 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:58.252168 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.252188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.253027 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.253031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.254059 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.254069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.254605 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.254609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.254795 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.255455 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:58.255457 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:58.255499 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:58.255571 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:58.255881 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:58.255890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:58.255896 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.256050 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.256363 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":158,"threads_started":1}
I20260924 10:38:58.258490 22887 heartbeater.cc:347] Connected to a master server at 127.2.193.190:33121
I20260924 10:38:58.258781 22887 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:58.259511 22887 heartbeater.cc:511] Master 127.2.193.190:33121 requested a full tablet report, sending...
I20260924 10:38:58.260756 22568 ts_manager.cc:194] Registered new tserver with Master: f2ae1882d7ee42198ed66ec9486c884b (127.2.193.130:34573)
I20260924 10:38:58.261413 22568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.130:56229
W20260924 10:38:58.369789 22890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:58.370091 22890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:58.370167 22890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:58.374315 22890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:58.374375 22890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:58.374468 22890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.193.131
I20260924 10:38:58.379606 22890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.193.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.193.131
--webserver_port=0
--tserver_master_addrs=127.2.193.190:33121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.193.131
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:58.380738 22890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:58.382198 22890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:58.391903 22896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:58.392102 22890 server_base.cc:1034] running on GCE node
W20260924 10:38:58.391993 22895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:58.391970 22898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:58.392670 22890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:58.393406 22890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:58.394595 22890 hybrid_clock.cc:648] HybridClock initialized: now 1790246338394575 us; error 32 us; skew 500 ppm
I20260924 10:38:58.396411 22890 webserver.cc:460] Webserver started at http://127.2.193.131:46455/ using document root <none> and password file <none>
I20260924 10:38:58.397037 22890 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:58.397135 22890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:58.397387 22890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:38:58.399478 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data/instance:
uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "f6568255e24018bccaefa3878007a2f0"
server_key_iv: "888c10dd8a803a384d18ca2d9b68bfe6"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.400178 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance:
uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "f6568255e24018bccaefa3878007a2f0"
server_key_iv: "888c10dd8a803a384d18ca2d9b68bfe6"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.404192 22890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 10:38:58.406731 22904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.408051 22890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:38:58.408183 22890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal
uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da"
format_stamp: "Formatted at 2026-09-24 10:38:58 on dist-test-slave-8wc7"
server_key: "f6568255e24018bccaefa3878007a2f0"
server_key_iv: "888c10dd8a803a384d18ca2d9b68bfe6"
server_key_version: "encryptionkey@0"
I20260924 10:38:58.408298 22890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:38:58.423365 22890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:38:58.424057 22890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:38:58.424245 22890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:38:58.424839 22890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:38:58.425865 22890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:38:58.425937 22890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.426014 22890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:38:58.426065 22890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:38:58.434751 22890 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.131:39869
I20260924 10:38:58.434768 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.131:39869 every 8 connection(s)
I20260924 10:38:58.435673 22890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/data/info.pb
I20260924 10:38:58.439667 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 22890
I20260924 10:38:58.439795 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal/instance
I20260924 10:38:58.440135 2822 external_mini_cluster.cc:1428] Setting key dc7ca87fc86a3296e0c589adaa2d88da
I20260924 10:38:58.449427 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.437510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.131:38657 (local address 127.2.193.190:33121)
0924 10:38:58.437660 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.437664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.438702 (+ 1038us) server_negotiation.cc:407] Connection header received
0924 10:38:58.440473 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.440499 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.440562 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.440654 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:58.441931 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.442682 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.443843 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.444067 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.444154 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.448914 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:58.448917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:58.448919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:58.448934 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:58.448957 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:58.448965 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:58.449033 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.449238 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.449302 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:38:58.450393 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.437876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:33121 (local address 127.2.193.131:38657)
0924 10:38:58.438533 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.438578 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.440186 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.440869 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.440877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.441000 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:38:58.441762 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.441781 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.442805 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.442808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.443718 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.443728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.444200 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.444202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.444349 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.448599 (+ 4250us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:38:58.448601 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:38:58.448653 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:38:58.448733 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:38:58.449049 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:38:58.449056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:38:58.449061 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.449166 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.449487 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260924 10:38:58.451716 23018 heartbeater.cc:347] Connected to a master server at 127.2.193.190:33121
I20260924 10:38:58.452158 23018 heartbeater.cc:464] Registering TS with master...
I20260924 10:38:58.452828 23018 heartbeater.cc:511] Master 127.2.193.190:33121 requested a full tablet report, sending...
I20260924 10:38:58.454121 22568 ts_manager.cc:194] Registered new tserver with Master: 1fd7215e5bb649e2a23dc9aebdd8f7da (127.2.193.131:39869)
I20260924 10:38:58.454723 2822 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:38:58.454834 22568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.193.131:38657
I20260924 10:38:58.460623 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.456950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.2.193.190:33121)
0924 10:38:58.457092 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.457096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.457302 (+ 206us) server_negotiation.cc:407] Connection header received
0924 10:38:58.457397 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.457399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.457436 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.457526 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:38:58.457835 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.458344 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.459166 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.459377 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.459456 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.460248 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:38:58.460251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:38:58.460252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:38:58.460264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:38:58.460281 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:38:58.460289 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:38:58.460348 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.460441 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.460507 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:38:58.463487 22568 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:38:58.466703 22568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:38:58.481848 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.475585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34573 (local address 127.0.0.1:58052)
0924 10:38:58.475989 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.476047 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.476172 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.477141 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.477146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.477166 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:58.477387 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.477396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.478917 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.478920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.480328 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.480337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.481139 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.481142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.481357 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.481409 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.481561 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.481683 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":104,"thread_start_us":110,"threads_started":1}
I20260924 10:38:58.481853 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.475744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.2.193.130:34573)
0924 10:38:58.476577 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.476593 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.476618 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:38:58.476699 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.476706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.476966 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.477571 (+ 605us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:58.477732 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.478795 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.480455 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.481000 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.481127 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.481370 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.481594 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.481693 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":121,"threads_started":1}
I20260924 10:38:58.482144 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.476610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:39869 (local address 127.0.0.1:48390)
0924 10:38:58.476912 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.476926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.477024 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.477916 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.477922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.477940 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:58.478056 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.478062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.479551 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.479558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.480621 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.480632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.481726 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.481729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.481845 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.481868 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.481914 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.481992 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":54,"thread_start_us":142,"threads_started":1}
I20260924 10:38:58.483345 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.476793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.2.193.131:39869)
0924 10:38:58.477403 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.477421 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.477449 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:38:58.477542 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.477547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.477755 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.478202 (+ 447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:58.478334 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.479423 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.480758 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.481598 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.482837 (+ 1239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.483038 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.483086 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.483178 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":149,"threads_started":1}
I20260924 10:38:58.483422 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.477352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:40281 (local address 127.0.0.1:52058)
0924 10:38:58.477917 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.477931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.478038 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.478468 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.478471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.478488 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:58.478582 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.478588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.480102 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.480106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.481314 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.481324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.483055 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.483059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.483157 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.483179 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.483241 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.483301 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260924 10:38:58.483521 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.477529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.2.193.129:40281)
0924 10:38:58.477890 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.477902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.478020 (+ 118us) server_negotiation.cc:407] Connection header received
0924 10:38:58.478146 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.478150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.478314 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.478465 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:58.478844 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.479968 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.481453 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.482948 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.483060 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.483220 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.483270 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.483353 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":125,"threads_started":1}
I20260924 10:38:58.484957 22821 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1d0c67f0f04c939883cf68017e3b1c (DEFAULT_TABLE table=test-table [id=53c03ce6705c433ea22a155fb7a3f5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:58.486147 22821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1d0c67f0f04c939883cf68017e3b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:58.485917 22952 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1d0c67f0f04c939883cf68017e3b1c (DEFAULT_TABLE table=test-table [id=53c03ce6705c433ea22a155fb7a3f5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:58.486102 22690 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1d0c67f0f04c939883cf68017e3b1c (DEFAULT_TABLE table=test-table [id=53c03ce6705c433ea22a155fb7a3f5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:38:58.486891 22952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1d0c67f0f04c939883cf68017e3b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:58.487069 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1d0c67f0f04c939883cf68017e3b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:38:58.492591 23035 tablet_bootstrap.cc:492] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Bootstrap starting.
I20260924 10:38:58.494285 23036 tablet_bootstrap.cc:492] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Bootstrap starting.
I20260924 10:38:58.494285 23037 tablet_bootstrap.cc:492] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb: Bootstrap starting.
I20260924 10:38:58.495177 23035 tablet_bootstrap.cc:654] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:58.495910 23035 log.cc:824] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:58.496455 23037 tablet_bootstrap.cc:654] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:58.496455 23036 tablet_bootstrap.cc:654] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Neither blocks nor log segments found. Creating new log.
I20260924 10:38:58.497350 23036 log.cc:824] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:58.497350 23037 log.cc:824] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:38:58.498028 23035 tablet_bootstrap.cc:492] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: No bootstrap required, opened a new log
I20260924 10:38:58.498234 23035 ts_tablet_manager.cc:1397] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:38:58.499678 23037 tablet_bootstrap.cc:492] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb: No bootstrap required, opened a new log
I20260924 10:38:58.499827 23036 tablet_bootstrap.cc:492] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: No bootstrap required, opened a new log
I20260924 10:38:58.499902 23037 ts_tablet_manager.cc:1397] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 10:38:58.500005 23036 ts_tablet_manager.cc:1397] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 10:38:58.502703 23035 raft_consensus.cc:348] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.502978 23035 raft_consensus.cc:374] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:58.503062 23035 raft_consensus.cc:729] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd7215e5bb649e2a23dc9aebdd8f7da, State: Initialized, Role: FOLLOWER
I20260924 10:38:58.502925 23037 raft_consensus.cc:348] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.503228 23037 raft_consensus.cc:374] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:58.503374 23037 raft_consensus.cc:729] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776b22c129f246f2b1686f939adbbbbb, State: Initialized, Role: FOLLOWER
I20260924 10:38:58.503508 23035 consensus_queue.cc:260] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.504201 23018 heartbeater.cc:503] Master 127.2.193.190:33121 was elected leader, sending a full tablet report...
I20260924 10:38:58.504096 23036 raft_consensus.cc:348] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.504317 23036 raft_consensus.cc:374] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:38:58.504361 23036 raft_consensus.cc:729] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ae1882d7ee42198ed66ec9486c884b, State: Initialized, Role: FOLLOWER
I20260924 10:38:58.504571 23035 ts_tablet_manager.cc:1428] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:38:58.504703 23037 consensus_queue.cc:260] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.504762 23036 consensus_queue.cc:260] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.505452 22887 heartbeater.cc:503] Master 127.2.193.190:33121 was elected leader, sending a full tablet report...
I20260924 10:38:58.505699 22756 heartbeater.cc:503] Master 127.2.193.190:33121 was elected leader, sending a full tablet report...
I20260924 10:38:58.505784 23036 ts_tablet_manager.cc:1428] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 10:38:58.506752 23037 ts_tablet_manager.cc:1428] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
W20260924 10:38:58.555259 22757 tablet.cc:2369] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:38:58.595539 23042 raft_consensus.cc:482] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:38:58.595804 23042 raft_consensus.cc:504] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.597223 23042 leader_election.cc:288] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2ae1882d7ee42198ed66ec9486c884b (127.2.193.130:34573), 1fd7215e5bb649e2a23dc9aebdd8f7da (127.2.193.131:39869)
I20260924 10:38:58.601643 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.597827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:49227 (local address 127.2.193.130:34573)
0924 10:38:58.598008 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.598012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.598079 (+ 67us) server_negotiation.cc:407] Connection header received
0924 10:38:58.598219 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.598222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.598290 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.598371 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:58.598718 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.599455 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.600622 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.601037 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.601172 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.601260 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.601421 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.601511 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 10:38:58.601672 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.597804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.193.129:58697 (local address 127.2.193.131:39869)
0924 10:38:58.597947 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.597953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.598333 (+ 380us) server_negotiation.cc:407] Connection header received
0924 10:38:58.598439 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.598442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.598491 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.598575 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:38:58.598944 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.599442 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.600615 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.601043 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.601208 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.601259 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.601482 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.601554 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:38:58.602074 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.597676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.131:39869 (local address 127.2.193.129:58697)
0924 10:38:58.598232 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.598247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.598348 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.598667 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.598672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.598695 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:58.598821 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.598827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.599548 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.599553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.600482 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.600491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.601223 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.601227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.601332 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.601349 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.601450 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.601513 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":262,"thread_start_us":102,"threads_started":1}
I20260924 10:38:58.602089 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.597676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.130:34573 (local address 127.2.193.129:49227)
0924 10:38:58.597961 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.597979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:38:58.598084 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:38:58.598397 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:38:58.598400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:38:58.598421 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:38:58.598576 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.598584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.599593 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.599595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.600477 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:38:58.600486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.601165 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.601190 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:38:58.601284 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.601315 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:38:58.601379 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:38:58.601474 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":28,"thread_start_us":80,"threads_started":1}
I20260924 10:38:58.602809 22972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d0c67f0f04c939883cf68017e3b1c" candidate_uuid: "776b22c129f246f2b1686f939adbbbbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" is_pre_election: true
I20260924 10:38:58.602897 22841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d0c67f0f04c939883cf68017e3b1c" candidate_uuid: "776b22c129f246f2b1686f939adbbbbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ae1882d7ee42198ed66ec9486c884b" is_pre_election: true
I20260924 10:38:58.603150 22972 raft_consensus.cc:2454] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 776b22c129f246f2b1686f939adbbbbb in term 0.
I20260924 10:38:58.603245 22841 raft_consensus.cc:2454] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 776b22c129f246f2b1686f939adbbbbb in term 0.
I20260924 10:38:58.603657 22643 leader_election.cc:302] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd7215e5bb649e2a23dc9aebdd8f7da, 776b22c129f246f2b1686f939adbbbbb; no voters:
I20260924 10:38:58.603952 23042 raft_consensus.cc:2781] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:38:58.604041 23042 raft_consensus.cc:482] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:38:58.604084 23042 raft_consensus.cc:3037] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:58.605108 23042 raft_consensus.cc:504] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.605556 23042 leader_election.cc:288] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [CANDIDATE]: Term 1 election: Requested vote from peers f2ae1882d7ee42198ed66ec9486c884b (127.2.193.130:34573), 1fd7215e5bb649e2a23dc9aebdd8f7da (127.2.193.131:39869)
I20260924 10:38:58.605955 22841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d0c67f0f04c939883cf68017e3b1c" candidate_uuid: "776b22c129f246f2b1686f939adbbbbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2ae1882d7ee42198ed66ec9486c884b"
I20260924 10:38:58.606076 22841 raft_consensus.cc:3037] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:58.607223 22841 raft_consensus.cc:2454] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 776b22c129f246f2b1686f939adbbbbb in term 1.
I20260924 10:38:58.607578 22646 leader_election.cc:302] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 776b22c129f246f2b1686f939adbbbbb, f2ae1882d7ee42198ed66ec9486c884b; no voters:
I20260924 10:38:58.607558 22972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1d0c67f0f04c939883cf68017e3b1c" candidate_uuid: "776b22c129f246f2b1686f939adbbbbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da"
I20260924 10:38:58.607694 22972 raft_consensus.cc:3037] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:38:58.607792 23042 raft_consensus.cc:2781] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:38:58.608050 23042 raft_consensus.cc:686] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [term 1 LEADER]: Becoming Leader. State: Replica: 776b22c129f246f2b1686f939adbbbbb, State: Running, Role: LEADER
I20260924 10:38:58.608388 23042 consensus_queue.cc:237] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } }
I20260924 10:38:58.608940 22972 raft_consensus.cc:2454] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 776b22c129f246f2b1686f939adbbbbb in term 1.
I20260924 10:38:58.611853 22568 catalog_manager.cc:5495] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 776b22c129f246f2b1686f939adbbbbb (127.2.193.129). New cstate: current_term: 1 leader_uuid: "776b22c129f246f2b1686f939adbbbbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "776b22c129f246f2b1686f939adbbbbb" member_type: VOTER last_known_addr { host: "127.2.193.129" port: 40281 } health_report { overall_health: HEALTHY } } }
I20260924 10:38:58.640178 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.636941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.2.193.129:40281)
0924 10:38:58.637085 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:38:58.637089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:38:58.637102 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:38:58.637225 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:38:58.637227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:38:58.637287 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:38:58.637372 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:38:58.637597 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.638088 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.638893 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:38:58.639092 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:38:58.639167 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:38:58.639429 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:38:58.639862 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:38:58.639953 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:38:58.640016 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 10:38:58.640061 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:38:58.646303 22972 raft_consensus.cc:1261] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 1 FOLLOWER]: Refusing update from remote peer 776b22c129f246f2b1686f939adbbbbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:58.646303 22841 raft_consensus.cc:1261] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 1 FOLLOWER]: Refusing update from remote peer 776b22c129f246f2b1686f939adbbbbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:38:58.647076 23046 consensus_queue.cc:1035] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd7215e5bb649e2a23dc9aebdd8f7da" member_type: VOTER last_known_addr { host: "127.2.193.131" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:58.647281 23042 consensus_queue.cc:1035] T 5c1d0c67f0f04c939883cf68017e3b1c P 776b22c129f246f2b1686f939adbbbbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2ae1882d7ee42198ed66ec9486c884b" member_type: VOTER last_known_addr { host: "127.2.193.130" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:38:58.655092 23051 mvcc.cc:205] Tried to move back new op lower bound from 7332849003087396864 to 7332849002943025152. Current Snapshot: MvccSnapshot[applied={T|T < 7332849003087396864}]
I20260924 10:38:58.666838 22567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:58.667007 22567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:38:58.669638 22567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e1bf121ced14b71828f283ece4a1fd7: Sending DeleteTablet for 3 replicas of tablet 5c1d0c67f0f04c939883cf68017e3b1c
I20260924 10:38:58.670472 22821 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1d0c67f0f04c939883cf68017e3b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:58 UTC) from {username='slave'} at 127.0.0.1:58052
I20260924 10:38:58.671026 23059 tablet_replica.cc:330] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: stopping tablet replica
I20260924 10:38:58.671072 22952 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1d0c67f0f04c939883cf68017e3b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:58 UTC) from {username='slave'} at 127.0.0.1:48390
I20260924 10:38:58.671242 22690 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1d0c67f0f04c939883cf68017e3b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:38:58 UTC) from {username='slave'} at 127.0.0.1:52058
I20260924 10:38:58.671535 23060 tablet_replica.cc:330] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: stopping tablet replica
I20260924 10:38:58.671590 23059 raft_consensus.cc:2229] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:58.671729 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 22628
I20260924 10:38:58.671823 23060 raft_consensus.cc:2229] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:38:58.672101 23060 raft_consensus.cc:2258] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:58.672791 23059 raft_consensus.cc:2258] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:38:58.673869 23060 ts_tablet_manager.cc:1905] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:58.674384 23059 ts_tablet_manager.cc:1905] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:38:58.676828 23060 ts_tablet_manager.cc:1918] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:58.676941 23060 log.cc:1196] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-2/wal/wals/5c1d0c67f0f04c939883cf68017e3b1c
I20260924 10:38:58.677064 23059 ts_tablet_manager.cc:1918] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:38:58.677162 23059 log.cc:1196] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Deleting WAL directory at /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246085895225-2822-0/minicluster-data/ts-1/wal/wals/5c1d0c67f0f04c939883cf68017e3b1c
I20260924 10:38:58.677316 23060 ts_tablet_manager.cc:1939] T 5c1d0c67f0f04c939883cf68017e3b1c P 1fd7215e5bb649e2a23dc9aebdd8f7da: Deleting consensus metadata
I20260924 10:38:58.677477 23059 ts_tablet_manager.cc:1939] T 5c1d0c67f0f04c939883cf68017e3b1c P f2ae1882d7ee42198ed66ec9486c884b: Deleting consensus metadata
I20260924 10:38:58.678689 22552 catalog_manager.cc:4841] TS 1fd7215e5bb649e2a23dc9aebdd8f7da (127.2.193.131:39869): tablet 5c1d0c67f0f04c939883cf68017e3b1c (table test-table [id=53c03ce6705c433ea22a155fb7a3f5f8]) successfully deleted
I20260924 10:38:58.678830 22555 catalog_manager.cc:4841] TS f2ae1882d7ee42198ed66ec9486c884b (127.2.193.130:34573): tablet 5c1d0c67f0f04c939883cf68017e3b1c (table test-table [id=53c03ce6705c433ea22a155fb7a3f5f8]) successfully deleted
W20260924 10:38:58.679096 22553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.193.129:40281 (error 108)
I20260924 10:38:58.679311 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 22759
I20260924 10:38:58.681054 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:38:58.680762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.129:40281 (local address 127.0.0.1:52072)
0924 10:38:58.680905 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:38:58.680980 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.129:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 10:38:58.681273 22553 catalog_manager.cc:4584] TS 776b22c129f246f2b1686f939adbbbbb (127.2.193.129:40281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c1d0c67f0f04c939883cf68017e3b1c: Network error: Client connection negotiation failed: client connection to 127.2.193.129:40281: connect: Connection refused (error 111)
I20260924 10:38:58.685159 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 22890
I20260924 10:38:58.690703 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 22537
2026-09-24T10:38:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1105 ms)
[----------] 6 tests from EncryptionPolicyTest (10436 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:38:58 dist-test-slave-8wc7 krb5kdc[23065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:38:58 dist-test-slave-8wc7 krb5kdc[23065](info): set up 2 sockets
Sep 24 10:38:58 dist-test-slave-8wc7 krb5kdc[23065](info): commencing operation
krb5kdc: starting...
W20260924 10:38:59.729384 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:38:59 dist-test-slave-8wc7 krb5kdc[23065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:38:59.857455 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:38:59.970116 23078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:38:59.970422 23078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:38:59.970503 23078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:38:59.974522 23078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:38:59.974628 23078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:38:59.974666 23078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:38:59.974699 23078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:38:59.974725 23078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:38:59.980036 23078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:38:59.981127 23078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:38:59.982197 23078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:38:59.991029 23083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:59.991080 23078 server_base.cc:1034] running on GCE node
W20260924 10:38:59.991062 23086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:38:59.991029 23084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:38:59.991920 23078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:38:59.993146 23078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:38:59.994366 23078 hybrid_clock.cc:648] HybridClock initialized: now 1790246339994338 us; error 45 us; skew 500 ppm
Sep 24 10:38:59 dist-test-slave-8wc7 krb5kdc[23065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:38:59.997023 23078 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:38:59.997938 23078 webserver.cc:460] Webserver started at http://127.0.0.1:33647/ using document root <none> and password file <none>
I20260924 10:38:59.998505 23078 fs_manager.cc:359] Metadata directory not provided
I20260924 10:38:59.998605 23078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:38:59.998837 23078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:00.000933 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "6cbb0eef4d6d4924a0f97464d505a02d"
format_stamp: "Formatted at 2026-09-24 10:38:59 on dist-test-slave-8wc7"
server_key: "b240b0091873eb43ed5650c7f7a13a70"
server_key_iv: "4aab6c587712eb8517ef2e8ee8965b80"
server_key_version: "encryptionkey@0"
I20260924 10:39:00.001601 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "6cbb0eef4d6d4924a0f97464d505a02d"
format_stamp: "Formatted at 2026-09-24 10:38:59 on dist-test-slave-8wc7"
server_key: "b240b0091873eb43ed5650c7f7a13a70"
server_key_iv: "4aab6c587712eb8517ef2e8ee8965b80"
server_key_version: "encryptionkey@0"
I20260924 10:39:00.005573 23078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:00.008116 23093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:00.009497 23078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:00.009641 23078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "6cbb0eef4d6d4924a0f97464d505a02d"
format_stamp: "Formatted at 2026-09-24 10:38:59 on dist-test-slave-8wc7"
server_key: "b240b0091873eb43ed5650c7f7a13a70"
server_key_iv: "4aab6c587712eb8517ef2e8ee8965b80"
server_key_version: "encryptionkey@0"
I20260924 10:39:00.009763 23078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:00.022496 23078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:00.025586 23078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:00.025803 23078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:00.033012 23078 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36163
I20260924 10:39:00.033041 23145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36163 every 8 connection(s)
I20260924 10:39:00.034168 23078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:00.037036 23146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:00.042832 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d: Bootstrap starting.
I20260924 10:39:00.043813 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23078
I20260924 10:39:00.043915 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:00.044219 2822 external_mini_cluster.cc:1428] Setting key 986a9a233259c169c77c7aeddd8b105a
I20260924 10:39:00.045956 23146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:00.046888 23146 log.cc:824] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:00.049333 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d: No bootstrap required, opened a new log
I20260924 10:39:00.053589 23146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260924 10:39:00.053894 23146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 10:39:00 dist-test-slave-8wc7 krb5kdc[23065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:39:00.053961 23146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbb0eef4d6d4924a0f97464d505a02d, State: Initialized, Role: FOLLOWER
I20260924 10:39:00.054417 23146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260924 10:39:00.054602 23146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:00.054672 23146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:00.054773 23146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:00.056100 23146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260924 10:39:00.056499 23146 leader_election.cc:302] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cbb0eef4d6d4924a0f97464d505a02d; no voters:
I20260924 10:39:00.056871 23146 leader_election.cc:288] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:00.056994 23151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:00.057269 23151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [term 1 LEADER]: Becoming Leader. State: Replica: 6cbb0eef4d6d4924a0f97464d505a02d, State: Running, Role: LEADER
I20260924 10:39:00.057634 23151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260924 10:39:00.058333 23146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:00.058972 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cbb0eef4d6d4924a0f97464d505a02d. Latest consensus state: current_term: 1 leader_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } } }
I20260924 10:39:00.059109 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:00.059685 23152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbb0eef4d6d4924a0f97464d505a02d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } } }
I20260924 10:39:00.059799 23152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:00.060585 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:00.045465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48697 (local address 127.0.0.1:36163)
0924 10:39:00.045979 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:00.046000 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:00.046047 (+ 47us) server_negotiation.cc:407] Connection header received
0924 10:39:00.046967 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:00.046996 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:00.047467 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:00.047853 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:00.048530 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:00.050040 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:00.050935 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:00.051648 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:00.051806 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:00.054645 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:00.054653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:00.054671 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:00.054709 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:00.056742 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:00.057504 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:00.057510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:00.057516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:00.057663 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:00.058090 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:00.058094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:00.058095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:00.058655 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:00.058894 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:00.059571 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 10:39:00.059756 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":145,"threads_started":1}
I20260924 10:39:00.064239 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 10:39:00.064507 23166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:00.064581 23166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:00.064639 23158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:00.065577 23158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:00.074155 23158 catalog_manager.cc:1347] Generated new cluster ID: 502d5ad16a974cfabb498801d31c19ae
I20260924 10:39:00.074227 23158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:00.077082 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:00.067719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33275 (local address 127.0.0.1:36163)
0924 10:39:00.067861 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:00.067868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:00.067884 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:39:00.067924 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:00.067927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:00.067970 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:00.068066 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:00.068897 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:00.069768 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:00.070995 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:00.071345 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:00.071488 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:00.073292 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:00.073296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:00.073299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:00.073325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:00.075130 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:00.075730 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:00.075733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:00.075735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:00.075825 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:00.076189 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:00.076192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:00.076194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:00.076390 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:00.076518 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:00.076826 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:39:00.076928 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:39:00.082118 23158 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:00.083357 23158 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:00.094414 23158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cbb0eef4d6d4924a0f97464d505a02d: Generated new TSK 0
I20260924 10:39:00.095391 23158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:00.099782 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23078
W20260924 10:39:00.105108 23063 connection.cc:535] client connection to 127.0.0.1:36163 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36163): Connection reset by peer (error 104)
2026-09-24T10:39:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:00 dist-test-slave-8wc7 krb5kdc[23177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:00 dist-test-slave-8wc7 krb5kdc[23177](info): set up 2 sockets
Sep 24 10:39:00 dist-test-slave-8wc7 krb5kdc[23177](info): commencing operation
krb5kdc: starting...
W20260924 10:39:01.145328 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:01 dist-test-slave-8wc7 krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:01.268718 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:01.381601 23190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:01.381956 23190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:01.382040 23190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:01.386332 23190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:01.386441 23190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:01.386480 23190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:01.386518 23190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:01.386554 23190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:01.392063 23190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:01.393357 23190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:01.394634 23190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:01.403687 23198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:01.403692 23196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:01.403750 23190 server_base.cc:1034] running on GCE node
W20260924 10:39:01.403692 23195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:01.404599 23190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:01.405865 23190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:01.407073 23190 hybrid_clock.cc:648] HybridClock initialized: now 1790246341407051 us; error 32 us; skew 500 ppm
Sep 24 10:39:01 dist-test-slave-8wc7 krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:01.409978 23190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:01.410943 23190 webserver.cc:460] Webserver started at http://127.0.0.1:38155/ using document root <none> and password file <none>
I20260924 10:39:01.411531 23190 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:01.411626 23190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:01.411862 23190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:01.414049 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "0d64384a9dbb4251b2eb32232f69f75a"
format_stamp: "Formatted at 2026-09-24 10:39:01 on dist-test-slave-8wc7"
server_key: "3b50d6d7086adbd2516db8efb959daa0"
server_key_iv: "9744e745b756ff42424463186f98817a"
server_key_version: "encryptionkey@0"
I20260924 10:39:01.414646 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "0d64384a9dbb4251b2eb32232f69f75a"
format_stamp: "Formatted at 2026-09-24 10:39:01 on dist-test-slave-8wc7"
server_key: "3b50d6d7086adbd2516db8efb959daa0"
server_key_iv: "9744e745b756ff42424463186f98817a"
server_key_version: "encryptionkey@0"
I20260924 10:39:01.418507 23190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:01.420961 23205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:01.422254 23190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:01.422422 23190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "0d64384a9dbb4251b2eb32232f69f75a"
format_stamp: "Formatted at 2026-09-24 10:39:01 on dist-test-slave-8wc7"
server_key: "3b50d6d7086adbd2516db8efb959daa0"
server_key_iv: "9744e745b756ff42424463186f98817a"
server_key_version: "encryptionkey@0"
I20260924 10:39:01.422540 23190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:01.440306 23190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:01.444016 23190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:01.444274 23190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:01.451996 23190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37803
I20260924 10:39:01.452014 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37803 every 8 connection(s)
I20260924 10:39:01.453020 23190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:01.455763 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23190
I20260924 10:39:01.455916 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:01.456224 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:01.456247 2822 external_mini_cluster.cc:1428] Setting key 117afcfd2240f1f87b4792c59373f08a
I20260924 10:39:01.462949 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a: Bootstrap starting.
I20260924 10:39:01.465603 23258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:01.465845 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:01.457787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35889 (local address 127.0.0.1:37803)
0924 10:39:01.458263 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:01.458279 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:01.458317 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:39:01.459154 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:01.459175 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:01.459540 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:01.459909 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:01.460340 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:01.461426 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:01.462353 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:01.462845 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:01.463030 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:01.464010 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:01.464016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:01.464029 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:01.464432 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:01.464486 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:01.464509 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:01.464661 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:01.464852 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:39:01.464997 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":158,"threads_started":1}
I20260924 10:39:01.466559 23258 log.cc:824] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:01.468783 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a: No bootstrap required, opened a new log
I20260924 10:39:01.469051 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:01.473122 23258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64384a9dbb4251b2eb32232f69f75a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260924 10:39:01.473469 23258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:01.473528 23258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d64384a9dbb4251b2eb32232f69f75a, State: Initialized, Role: FOLLOWER
I20260924 10:39:01.474016 23258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64384a9dbb4251b2eb32232f69f75a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260924 10:39:01.474195 23258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:01.474272 23258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:01.474388 23258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:01.475407 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:01.471148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44629 (local address 127.0.0.1:37803)
0924 10:39:01.471286 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:01.471291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:01.471478 (+ 187us) server_negotiation.cc:407] Connection header received
0924 10:39:01.471591 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:01.471593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:01.471633 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:01.471698 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:01.472012 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:01.472496 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:01.473639 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:01.473940 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:01.474022 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:01.474948 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:01.474950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:01.474952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:01.474964 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:01.474987 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:01.474996 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:01.475067 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:01.475213 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 10:39:01.475243 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:39:01.475776 23258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64384a9dbb4251b2eb32232f69f75a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260924 10:39:01.476172 23258 leader_election.cc:302] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d64384a9dbb4251b2eb32232f69f75a; no voters:
I20260924 10:39:01.476545 23258 leader_election.cc:288] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:01.476655 23268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:01.476891 23268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [term 1 LEADER]: Becoming Leader. State: Replica: 0d64384a9dbb4251b2eb32232f69f75a, State: Running, Role: LEADER
I20260924 10:39:01.477293 23268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64384a9dbb4251b2eb32232f69f75a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } }
I20260924 10:39:01.477941 23258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:01.479030 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d64384a9dbb4251b2eb32232f69f75a. Latest consensus state: current_term: 1 leader_uuid: "0d64384a9dbb4251b2eb32232f69f75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64384a9dbb4251b2eb32232f69f75a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } } }
I20260924 10:39:01.479183 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:01.479034 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d64384a9dbb4251b2eb32232f69f75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64384a9dbb4251b2eb32232f69f75a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37803 } } }
I20260924 10:39:01.479532 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:01.479552 23277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:01.482842 23277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:01.488958 23277 catalog_manager.cc:1347] Generated new cluster ID: 8222f406ce144e479444109ccc20319b
I20260924 10:39:01.489046 23277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:01.505337 23277 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:01.506675 23277 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:01.520226 23277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d64384a9dbb4251b2eb32232f69f75a: Generated new TSK 0
I20260924 10:39:01.521044 23277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:01.528622 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23190
W20260924 10:39:01.534060 23175 connection.cc:535] client connection to 127.0.0.1:37803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37803): Connection reset by peer (error 104)
2026-09-24T10:39:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:01 dist-test-slave-8wc7 krb5kdc[23289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:01 dist-test-slave-8wc7 krb5kdc[23289](info): set up 2 sockets
Sep 24 10:39:01 dist-test-slave-8wc7 krb5kdc[23289](info): commencing operation
krb5kdc: starting...
W20260924 10:39:02.585567 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:02 dist-test-slave-8wc7 krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:02.711813 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:02.825413 23302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:02.825698 23302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:02.825742 23302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:02.829813 23302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:02.829877 23302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:02.829896 23302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:02.829913 23302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:02.829928 23302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:02.835273 23302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:02.836361 23302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:02.837545 23302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:02.846047 23310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:02.846040 23308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:02.846072 23307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:02.846189 23302 server_base.cc:1034] running on GCE node
I20260924 10:39:02.846936 23302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:02.848191 23302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:02.849387 23302 hybrid_clock.cc:648] HybridClock initialized: now 1790246342849360 us; error 49 us; skew 500 ppm
Sep 24 10:39:02 dist-test-slave-8wc7 krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:02.852124 23302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:02.853016 23302 webserver.cc:460] Webserver started at http://127.0.0.1:39447/ using document root <none> and password file <none>
I20260924 10:39:02.853657 23302 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:02.853752 23302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:02.853993 23302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:02.856084 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "31d3e58170484f87a23e49daef3807af"
format_stamp: "Formatted at 2026-09-24 10:39:02 on dist-test-slave-8wc7"
server_key: "ee4ffa5fec1d4a62549b017055529a5e"
server_key_iv: "a7faa56a7486547eb15cb098e8b33d96"
server_key_version: "encryptionkey@0"
I20260924 10:39:02.856670 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "31d3e58170484f87a23e49daef3807af"
format_stamp: "Formatted at 2026-09-24 10:39:02 on dist-test-slave-8wc7"
server_key: "ee4ffa5fec1d4a62549b017055529a5e"
server_key_iv: "a7faa56a7486547eb15cb098e8b33d96"
server_key_version: "encryptionkey@0"
I20260924 10:39:02.860646 23302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:39:02.863238 23317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:02.864495 23302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:39:02.864660 23302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "31d3e58170484f87a23e49daef3807af"
format_stamp: "Formatted at 2026-09-24 10:39:02 on dist-test-slave-8wc7"
server_key: "ee4ffa5fec1d4a62549b017055529a5e"
server_key_iv: "a7faa56a7486547eb15cb098e8b33d96"
server_key_version: "encryptionkey@0"
I20260924 10:39:02.864792 23302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:02.869827 23302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:02.872794 23302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:02.873018 23302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:02.880355 23302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44463
I20260924 10:39:02.880379 23369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44463 every 8 connection(s)
I20260924 10:39:02.881374 23302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:02.884192 23370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:02.888171 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23302
I20260924 10:39:02.888327 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:02.888612 2822 external_mini_cluster.cc:1428] Setting key c465d075c63760487eb12b5a7f78b074
I20260924 10:39:02.890702 23370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af: Bootstrap starting.
I20260924 10:39:02.893591 23370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:02.894491 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:02.890090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60943 (local address 127.0.0.1:44463)
0924 10:39:02.890549 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:02.890569 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:02.890623 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:39:02.891453 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:02.891478 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:02.891868 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:02.892268 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:02.893012 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:02.893018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:02.893032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:02.893044 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:02.893111 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:02.893136 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:02.893272 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:02.893372 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 10:39:02.893533 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260924 10:39:02.894500 23370 log.cc:824] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:02.896867 23370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af: No bootstrap required, opened a new log
I20260924 10:39:02.897495 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:02.901124 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:02.899496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47833 (local address 127.0.0.1:44463)
0924 10:39:02.899624 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:02.899627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:02.899860 (+ 233us) server_negotiation.cc:407] Connection header received
0924 10:39:02.899948 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:02.899952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:02.899999 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:02.900092 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:02.900784 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:02.900788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:02.900790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:02.900802 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:02.900829 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:02.900841 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:02.900937 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:02.900970 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 10:39:02.901006 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:39:02.901373 23370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3e58170484f87a23e49daef3807af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260924 10:39:02.901602 23370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:02.901667 23370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d3e58170484f87a23e49daef3807af, State: Initialized, Role: FOLLOWER
I20260924 10:39:02.902125 23370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3e58170484f87a23e49daef3807af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260924 10:39:02.902292 23370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:02.902365 23370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:02.902441 23370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:02.903388 23370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3e58170484f87a23e49daef3807af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260924 10:39:02.903764 23370 leader_election.cc:302] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d3e58170484f87a23e49daef3807af; no voters:
I20260924 10:39:02.904165 23370 leader_election.cc:288] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:02.904251 23380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:02.904448 23380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [term 1 LEADER]: Becoming Leader. State: Replica: 31d3e58170484f87a23e49daef3807af, State: Running, Role: LEADER
I20260924 10:39:02.904764 23380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3e58170484f87a23e49daef3807af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } }
I20260924 10:39:02.905582 23370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:02.906569 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d3e58170484f87a23e49daef3807af. Latest consensus state: current_term: 1 leader_uuid: "31d3e58170484f87a23e49daef3807af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3e58170484f87a23e49daef3807af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } } }
I20260924 10:39:02.906780 23381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d3e58170484f87a23e49daef3807af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d3e58170484f87a23e49daef3807af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44463 } } }
I20260924 10:39:02.906880 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:02.907002 23381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:02.907915 23389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:02.910820 23389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:02.916736 23389 catalog_manager.cc:1347] Generated new cluster ID: edfb3649ae884472b9878531409f134c
I20260924 10:39:02.916819 23389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:02.927475 23389 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:02.927887 23389 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:02.933315 23389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d3e58170484f87a23e49daef3807af: Generated new TSK 0
I20260924 10:39:02.934237 23389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:02.945504 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23302
2026-09-24T10:39:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:03 dist-test-slave-8wc7 krb5kdc[23401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:03 dist-test-slave-8wc7 krb5kdc[23401](info): set up 2 sockets
Sep 24 10:39:03 dist-test-slave-8wc7 krb5kdc[23401](info): commencing operation
krb5kdc: starting...
W20260924 10:39:03.985353 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:04 dist-test-slave-8wc7 krb5kdc[23401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:04.113051 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:04.226056 23414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:04.226395 23414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:04.226485 23414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:04.230618 23414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:04.230723 23414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:04.230762 23414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:04.230796 23414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:04.230824 23414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:04.236207 23414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:04.237401 23414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:04.238507 23414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:04.248580 23420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:04.248580 23419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:04.248956 23414 server_base.cc:1034] running on GCE node
W20260924 10:39:04.249197 23422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:04.249858 23414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:04.251067 23414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:04.252249 23414 hybrid_clock.cc:648] HybridClock initialized: now 1790246344252232 us; error 32 us; skew 500 ppm
Sep 24 10:39:04 dist-test-slave-8wc7 krb5kdc[23401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:04.261854 23414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:04.262841 23414 webserver.cc:460] Webserver started at http://127.0.0.1:38775/ using document root <none> and password file <none>
I20260924 10:39:04.263417 23414 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:04.263478 23414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:04.263653 23414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:04.266018 23414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "52148d1fa14e4c49b7c21a4f9c213193"
format_stamp: "Formatted at 2026-09-24 10:39:04 on dist-test-slave-8wc7"
server_key: "d0f0feaf4a5673a6e4e44e7cda7234e3"
server_key_iv: "0554b85f0ff6058e73a7c86f35eced6f"
server_key_version: "encryptionkey@0"
I20260924 10:39:04.266564 23414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "52148d1fa14e4c49b7c21a4f9c213193"
format_stamp: "Formatted at 2026-09-24 10:39:04 on dist-test-slave-8wc7"
server_key: "d0f0feaf4a5673a6e4e44e7cda7234e3"
server_key_iv: "0554b85f0ff6058e73a7c86f35eced6f"
server_key_version: "encryptionkey@0"
I20260924 10:39:04.270510 23414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:04.273051 23429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:04.274367 23414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:04.274487 23414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "52148d1fa14e4c49b7c21a4f9c213193"
format_stamp: "Formatted at 2026-09-24 10:39:04 on dist-test-slave-8wc7"
server_key: "d0f0feaf4a5673a6e4e44e7cda7234e3"
server_key_iv: "0554b85f0ff6058e73a7c86f35eced6f"
server_key_version: "encryptionkey@0"
I20260924 10:39:04.274585 23414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:04.302009 23414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:04.305038 23414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:04.305315 23414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:04.312598 23414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39615
I20260924 10:39:04.312634 23481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39615 every 8 connection(s)
I20260924 10:39:04.313673 23414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:04.316380 23482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:04.320304 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23414
I20260924 10:39:04.320479 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:04.320765 2822 external_mini_cluster.cc:1428] Setting key fadad485607c598ccece6456f0581ec9
I20260924 10:39:04.322420 23482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193: Bootstrap starting.
I20260924 10:39:04.324690 23482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:04.325538 23482 log.cc:824] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:04.327500 23482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193: No bootstrap required, opened a new log
Sep 24 10:39:04 dist-test-slave-8wc7 krb5kdc[23401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:39:04.331990 23482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52148d1fa14e4c49b7c21a4f9c213193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260924 10:39:04.332288 23482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:04.332381 23482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52148d1fa14e4c49b7c21a4f9c213193, State: Initialized, Role: FOLLOWER
I20260924 10:39:04.332870 23482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52148d1fa14e4c49b7c21a4f9c213193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260924 10:39:04.332994 23482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:04.333097 23482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:04.333235 23482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:04.334178 23482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52148d1fa14e4c49b7c21a4f9c213193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260924 10:39:04.334539 23482 leader_election.cc:302] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52148d1fa14e4c49b7c21a4f9c213193; no voters:
I20260924 10:39:04.334901 23482 leader_election.cc:288] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:04.334978 23487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:04.335201 23487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [term 1 LEADER]: Becoming Leader. State: Replica: 52148d1fa14e4c49b7c21a4f9c213193, State: Running, Role: LEADER
I20260924 10:39:04.335547 23487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52148d1fa14e4c49b7c21a4f9c213193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } }
I20260924 10:39:04.336292 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:04.322273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51575 (local address 127.0.0.1:39615)
0924 10:39:04.322738 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:04.322754 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:04.322793 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:39:04.323448 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:04.323465 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:04.323826 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:04.324194 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:04.325045 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:04.326366 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:04.327270 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:04.327837 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:04.328025 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:04.330992 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:04.330998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:04.331013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:04.331043 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:04.333037 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:04.333616 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:04.333623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:04.333629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:04.333759 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:04.334056 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:04.334060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:04.334062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:04.334549 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:04.334756 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:04.335088 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:39:04.335442 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":131,"threads_started":1}
I20260924 10:39:04.336295 23482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:04.336737 23489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52148d1fa14e4c49b7c21a4f9c213193. Latest consensus state: current_term: 1 leader_uuid: "52148d1fa14e4c49b7c21a4f9c213193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52148d1fa14e4c49b7c21a4f9c213193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } } }
I20260924 10:39:04.336726 23488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52148d1fa14e4c49b7c21a4f9c213193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52148d1fa14e4c49b7c21a4f9c213193" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39615 } } }
I20260924 10:39:04.336881 23488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:04.336881 23489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:04.337306 23491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:04.340006 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:04.340548 23491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:04.346401 23491 catalog_manager.cc:1347] Generated new cluster ID: 35275ba18e3e4ca0b2d9f841e431be2b
I20260924 10:39:04.346472 23491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:04.353930 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:04.345526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42371 (local address 127.0.0.1:39615)
0924 10:39:04.345651 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:04.345656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:04.346030 (+ 374us) server_negotiation.cc:407] Connection header received
0924 10:39:04.346234 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:04.346237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:04.346288 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:04.346340 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:04.347166 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:04.347754 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:04.348618 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:04.348887 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:04.349009 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:04.350585 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:04.350589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:04.350592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:04.350628 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:04.352158 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:04.352611 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:04.352615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:04.352617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:04.352701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:04.352989 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:04.352993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:04.352995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:04.353249 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:04.353391 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:04.353635 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 10:39:04.353761 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:39:04.365334 23491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:04.366487 23491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:04.374557 23491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52148d1fa14e4c49b7c21a4f9c213193: Generated new TSK 0
I20260924 10:39:04.375480 23491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:04.389609 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23414
2026-09-24T10:39:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:04 dist-test-slave-8wc7 krb5kdc[23513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:04 dist-test-slave-8wc7 krb5kdc[23513](info): set up 2 sockets
Sep 24 10:39:04 dist-test-slave-8wc7 krb5kdc[23513](info): commencing operation
krb5kdc: starting...
W20260924 10:39:05.425495 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:05 dist-test-slave-8wc7 krb5kdc[23513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:05.552383 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:05.666999 23526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:05.667366 23526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:05.667479 23526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:05.671629 23526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:05.671754 23526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:05.671793 23526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:05.671828 23526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:05.671860 23526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:05.677610 23526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:05.678814 23526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:05.679921 23526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:05.689365 23531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:05.689282 23532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:05.689589 23526 server_base.cc:1034] running on GCE node
W20260924 10:39:05.689720 23534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:05.690318 23526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:05.691519 23526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:05.692684 23526 hybrid_clock.cc:648] HybridClock initialized: now 1790246345692665 us; error 32 us; skew 500 ppm
Sep 24 10:39:05 dist-test-slave-8wc7 krb5kdc[23513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:05.695629 23526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:05.696650 23526 webserver.cc:460] Webserver started at http://127.0.0.1:36221/ using document root <none> and password file <none>
I20260924 10:39:05.697319 23526 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:05.697423 23526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:05.697683 23526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:05.699867 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "9794bd36fead47d7bba39e63479fc424"
format_stamp: "Formatted at 2026-09-24 10:39:05 on dist-test-slave-8wc7"
server_key: "c83c869c0f1a141af22a537f34562cf1"
server_key_iv: "690ac776e4ed66772ca8723d62d446ff"
server_key_version: "encryptionkey@0"
I20260924 10:39:05.700484 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "9794bd36fead47d7bba39e63479fc424"
format_stamp: "Formatted at 2026-09-24 10:39:05 on dist-test-slave-8wc7"
server_key: "c83c869c0f1a141af22a537f34562cf1"
server_key_iv: "690ac776e4ed66772ca8723d62d446ff"
server_key_version: "encryptionkey@0"
I20260924 10:39:05.704483 23526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:39:05.707111 23541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:05.708343 23526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:39:05.708501 23526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "9794bd36fead47d7bba39e63479fc424"
format_stamp: "Formatted at 2026-09-24 10:39:05 on dist-test-slave-8wc7"
server_key: "c83c869c0f1a141af22a537f34562cf1"
server_key_iv: "690ac776e4ed66772ca8723d62d446ff"
server_key_version: "encryptionkey@0"
I20260924 10:39:05.708637 23526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:05.721863 23526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:05.725566 23526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:05.725837 23526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:05.733420 23526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36907
I20260924 10:39:05.733444 23593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36907 every 8 connection(s)
I20260924 10:39:05.734464 23526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:05.737437 23594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:05.739255 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23526
I20260924 10:39:05.739405 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:05.739718 2822 external_mini_cluster.cc:1428] Setting key e216acb625303e30d80079551e7c06db
I20260924 10:39:05.744210 23594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424: Bootstrap starting.
I20260924 10:39:05.746699 23594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:05.747524 23594 log.cc:824] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:05.749302 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:05.741338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57037 (local address 127.0.0.1:36907)
0924 10:39:05.741822 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:05.741839 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:05.741877 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:39:05.742565 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:05.742584 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:05.742952 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:05.743328 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:05.743750 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:05.744822 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:05.745845 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:05.746344 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:05.746535 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:05.747460 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:05.747465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:05.747479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:05.747937 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:05.747994 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:05.748018 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:05.748145 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:05.748263 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 10:39:05.748480 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":147,"threads_started":1}
I20260924 10:39:05.749867 23594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424: No bootstrap required, opened a new log
I20260924 10:39:05.752274 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:05.754014 23594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9794bd36fead47d7bba39e63479fc424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36907 } }
I20260924 10:39:05.754303 23594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:05.754424 23594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9794bd36fead47d7bba39e63479fc424, State: Initialized, Role: FOLLOWER
I20260924 10:39:05.754976 23594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9794bd36fead47d7bba39e63479fc424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36907 } }
I20260924 10:39:05.755156 23594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:05.755229 23594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:05.755332 23594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:05.756510 23594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9794bd36fead47d7bba39e63479fc424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36907 } }
I20260924 10:39:05.756871 23594 leader_election.cc:302] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9794bd36fead47d7bba39e63479fc424; no voters:
I20260924 10:39:05.757265 23594 leader_election.cc:288] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:05.757375 23604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:05.757612 23604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [term 1 LEADER]: Becoming Leader. State: Replica: 9794bd36fead47d7bba39e63479fc424, State: Running, Role: LEADER
I20260924 10:39:05.757973 23604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9794bd36fead47d7bba39e63479fc424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36907 } }
I20260924 10:39:05.758798 23594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:05.758976 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:05.754443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44069 (local address 127.0.0.1:36907)
0924 10:39:05.754587 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:05.754591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:05.754822 (+ 231us) server_negotiation.cc:407] Connection header received
0924 10:39:05.754902 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:05.754904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:05.754952 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:05.755042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:05.755354 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:05.756098 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:05.756968 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:05.757391 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:05.757542 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:05.758517 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:05.758521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:05.758523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:05.758538 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:05.758569 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:05.758582 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:05.758670 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:05.758795 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 10:39:05.758848 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:39:05.759744 23605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9794bd36fead47d7bba39e63479fc424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9794bd36fead47d7bba39e63479fc424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36907 } } }
I20260924 10:39:05.759879 23605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:05.760308 23613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:05.760325 23606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9794bd36fead47d7bba39e63479fc424. Latest consensus state: current_term: 1 leader_uuid: "9794bd36fead47d7bba39e63479fc424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9794bd36fead47d7bba39e63479fc424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36907 } } }
I20260924 10:39:05.760468 23606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:05.763635 23613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:05.769582 23613 catalog_manager.cc:1347] Generated new cluster ID: f8f1d0a1598f4bb68c74a2f2327255f5
I20260924 10:39:05.769671 23613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:05.782176 23613 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:05.783221 23613 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:05.795013 23613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9794bd36fead47d7bba39e63479fc424: Generated new TSK 0
I20260924 10:39:05.795977 23613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:05.807758 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23526
2026-09-24T10:39:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:05 dist-test-slave-8wc7 krb5kdc[23625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:05 dist-test-slave-8wc7 krb5kdc[23625](info): set up 2 sockets
Sep 24 10:39:05 dist-test-slave-8wc7 krb5kdc[23625](info): commencing operation
krb5kdc: starting...
W20260924 10:39:06.845515 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:06 dist-test-slave-8wc7 krb5kdc[23625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:06.972414 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:07.088354 23638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:07.088724 23638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:07.088815 23638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:07.093331 23638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:07.093473 23638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:07.093511 23638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:07.093564 23638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:07.093601 23638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:07.099509 23638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:07.101056 23638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:07.102515 23638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:07.111510 23646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:07.111526 23643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:07.111523 23644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:07.111792 23638 server_base.cc:1034] running on GCE node
I20260924 10:39:07.112390 23638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:07.113652 23638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:07.114845 23638 hybrid_clock.cc:648] HybridClock initialized: now 1790246347114810 us; error 49 us; skew 500 ppm
Sep 24 10:39:07 dist-test-slave-8wc7 krb5kdc[23625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:07.117712 23638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:07.118736 23638 webserver.cc:460] Webserver started at http://127.0.0.1:37911/ using document root <none> and password file <none>
I20260924 10:39:07.119333 23638 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:07.119431 23638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:07.119671 23638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:07.121901 23638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "1883aa84756b436db00a75bfb31d73c3"
format_stamp: "Formatted at 2026-09-24 10:39:07 on dist-test-slave-8wc7"
server_key: "ed04d1573a5f79af5f519c523d7b9391"
server_key_iv: "50faece85f766eaaa2199a45d4be5b34"
server_key_version: "encryptionkey@0"
I20260924 10:39:07.122488 23638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "1883aa84756b436db00a75bfb31d73c3"
format_stamp: "Formatted at 2026-09-24 10:39:07 on dist-test-slave-8wc7"
server_key: "ed04d1573a5f79af5f519c523d7b9391"
server_key_iv: "50faece85f766eaaa2199a45d4be5b34"
server_key_version: "encryptionkey@0"
I20260924 10:39:07.126550 23638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:07.129745 23653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:07.131233 23638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:39:07.131403 23638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "1883aa84756b436db00a75bfb31d73c3"
format_stamp: "Formatted at 2026-09-24 10:39:07 on dist-test-slave-8wc7"
server_key: "ed04d1573a5f79af5f519c523d7b9391"
server_key_iv: "50faece85f766eaaa2199a45d4be5b34"
server_key_version: "encryptionkey@0"
I20260924 10:39:07.131548 23638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:07.136629 23638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:07.139967 23638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:07.140183 23638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:07.148518 23638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37077
I20260924 10:39:07.148547 23705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37077 every 8 connection(s)
I20260924 10:39:07.149654 23638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:07.152544 23706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:07.158843 23706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3: Bootstrap starting.
I20260924 10:39:07.159142 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23638
I20260924 10:39:07.159287 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:07.159579 2822 external_mini_cluster.cc:1428] Setting key c72efb7d10755385757bb6781751b9bb
I20260924 10:39:07.162093 23706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:07.163043 23706 log.cc:824] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:07.165570 23706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3: No bootstrap required, opened a new log
I20260924 10:39:07.165915 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:07.161017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52357 (local address 127.0.0.1:37077)
0924 10:39:07.161573 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:07.161592 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:07.161636 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:39:07.162615 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:07.162641 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:07.163033 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:07.163439 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:07.164411 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:07.164416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:07.164430 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:07.164440 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:07.164501 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:07.164521 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:07.164638 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:07.164775 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 10:39:07.164976 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":195,"threads_started":1}
I20260924 10:39:07.169027 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:07.169965 23706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883aa84756b436db00a75bfb31d73c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37077 } }
I20260924 10:39:07.170231 23706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:07.170286 23706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1883aa84756b436db00a75bfb31d73c3, State: Initialized, Role: FOLLOWER
I20260924 10:39:07.170763 23706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883aa84756b436db00a75bfb31d73c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37077 } }
I20260924 10:39:07.170917 23706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:07.171005 23706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:07.171128 23706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:07.172122 23706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883aa84756b436db00a75bfb31d73c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37077 } }
I20260924 10:39:07.172489 23706 leader_election.cc:302] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1883aa84756b436db00a75bfb31d73c3; no voters:
I20260924 10:39:07.172794 23706 leader_election.cc:288] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:07.172879 23716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:07.173110 23716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1883aa84756b436db00a75bfb31d73c3, State: Running, Role: LEADER
I20260924 10:39:07.173310 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:07.171263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56903 (local address 127.0.0.1:37077)
0924 10:39:07.171395 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:07.171400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:07.171694 (+ 294us) server_negotiation.cc:407] Connection header received
0924 10:39:07.171855 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:07.171859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:07.171898 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:07.171981 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:07.172746 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:07.172749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:07.172752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:07.172763 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:07.172794 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:07.172808 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:07.172891 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:07.173074 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 10:39:07.173118 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:39:07.173483 23716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883aa84756b436db00a75bfb31d73c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37077 } }
I20260924 10:39:07.174227 23706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:07.175278 23717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1883aa84756b436db00a75bfb31d73c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883aa84756b436db00a75bfb31d73c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37077 } } }
I20260924 10:39:07.175406 23717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:07.175729 23718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1883aa84756b436db00a75bfb31d73c3. Latest consensus state: current_term: 1 leader_uuid: "1883aa84756b436db00a75bfb31d73c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1883aa84756b436db00a75bfb31d73c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37077 } } }
I20260924 10:39:07.175819 23718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:07.175921 23725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:07.179111 23725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:07.185329 23725 catalog_manager.cc:1347] Generated new cluster ID: a7779f6499264e86b5b9619841a98cc6
I20260924 10:39:07.185420 23725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:07.203140 23725 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:07.203552 23725 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:07.207587 23725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1883aa84756b436db00a75bfb31d73c3: Generated new TSK 0
I20260924 10:39:07.208380 23725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:07.220243 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23638
2026-09-24T10:39:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:07 dist-test-slave-8wc7 krb5kdc[23737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:07 dist-test-slave-8wc7 krb5kdc[23737](info): set up 2 sockets
Sep 24 10:39:07 dist-test-slave-8wc7 krb5kdc[23737](info): commencing operation
krb5kdc: starting...
W20260924 10:39:08.257412 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:08 dist-test-slave-8wc7 krb5kdc[23737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:08.380832 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:08.498322 23750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:08.498615 23750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:08.498703 23750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:08.502965 23750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:08.503041 23750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:08.503062 23750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:08.503078 23750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:08.503131 23750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:08.508596 23750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:08.509737 23750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:08.510833 23750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:08.519531 23756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:08.519533 23755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:08.519618 23750 server_base.cc:1034] running on GCE node
W20260924 10:39:08.519531 23758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:08.520356 23750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:08.521595 23750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:08.522791 23750 hybrid_clock.cc:648] HybridClock initialized: now 1790246348522776 us; error 27 us; skew 500 ppm
Sep 24 10:39:08 dist-test-slave-8wc7 krb5kdc[23737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:08.525527 23750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:08.526510 23750 webserver.cc:460] Webserver started at http://127.0.0.1:39923/ using document root <none> and password file <none>
I20260924 10:39:08.527087 23750 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:08.527192 23750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:08.527444 23750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:08.529604 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8"
format_stamp: "Formatted at 2026-09-24 10:39:08 on dist-test-slave-8wc7"
server_key: "c53a29dfed55385370ccf10029b02395"
server_key_iv: "59813850ca742f0493643ec68bfba466"
server_key_version: "encryptionkey@0"
I20260924 10:39:08.530185 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8"
format_stamp: "Formatted at 2026-09-24 10:39:08 on dist-test-slave-8wc7"
server_key: "c53a29dfed55385370ccf10029b02395"
server_key_iv: "59813850ca742f0493643ec68bfba466"
server_key_version: "encryptionkey@0"
I20260924 10:39:08.534075 23750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:39:08.536582 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:08.537894 23750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:08.538039 23750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8"
format_stamp: "Formatted at 2026-09-24 10:39:08 on dist-test-slave-8wc7"
server_key: "c53a29dfed55385370ccf10029b02395"
server_key_iv: "59813850ca742f0493643ec68bfba466"
server_key_version: "encryptionkey@0"
I20260924 10:39:08.538167 23750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:08.550033 23750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:08.553294 23750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:08.553522 23750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:08.560855 23750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33585
I20260924 10:39:08.560868 23817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33585 every 8 connection(s)
I20260924 10:39:08.561897 23750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:08.564822 23818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:08.567530 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23750
I20260924 10:39:08.567691 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:08.568020 2822 external_mini_cluster.cc:1428] Setting key ef1003f5c77f12795ae6db2a039a09bf
I20260924 10:39:08.571144 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8: Bootstrap starting.
I20260924 10:39:08.573899 23818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:08.574785 23818 log.cc:824] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:08.576970 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8: No bootstrap required, opened a new log
Sep 24 10:39:08 dist-test-slave-8wc7 krb5kdc[23737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:39:08.580933 23818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260924 10:39:08.581161 23818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:08.581314 23818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c69f1dc1ff44f7a682c3a8bc387cb8, State: Initialized, Role: FOLLOWER
I20260924 10:39:08.581867 23818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260924 10:39:08.582059 23818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:08.582134 23818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:08.582258 23818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:08.582950 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:08.569498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:43697 (local address 127.0.0.1:33585)
0924 10:39:08.569907 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:08.569935 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:08.569972 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:39:08.570597 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:08.570624 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:08.570988 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:08.571329 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:08.572168 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:08.573244 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:08.574419 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:08.574956 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:08.575084 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:08.577917 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:08.577923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:08.577937 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:08.577965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:08.579716 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:08.580266 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:08.580279 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:08.580284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:08.580399 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:08.580757 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:08.580767 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:08.580768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:08.581235 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:08.581491 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:08.581796 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 10:39:08.582059 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":142,"threads_started":1}
I20260924 10:39:08.583308 23818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260924 10:39:08.583668 23818 leader_election.cc:302] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50c69f1dc1ff44f7a682c3a8bc387cb8; no voters:
I20260924 10:39:08.583985 23818 leader_election.cc:288] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:08.584121 23823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:08.584347 23823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [term 1 LEADER]: Becoming Leader. State: Replica: 50c69f1dc1ff44f7a682c3a8bc387cb8, State: Running, Role: LEADER
I20260924 10:39:08.584717 23823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260924 10:39:08.585351 23818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:08.586400 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:08.586658 23825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50c69f1dc1ff44f7a682c3a8bc387cb8. Latest consensus state: current_term: 1 leader_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } } }
I20260924 10:39:08.586658 23824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50c69f1dc1ff44f7a682c3a8bc387cb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } } }
I20260924 10:39:08.586803 23824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:08.586803 23825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:08.587217 23833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:08.590941 23833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:08.596789 23833 catalog_manager.cc:1347] Generated new cluster ID: a562c3b76c0e4f46b1a698ed8989538e
I20260924 10:39:08.596918 23833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:08.598124 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:08.588750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:43793 (local address 127.0.0.1:33585)
0924 10:39:08.588864 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:08.588868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:08.589099 (+ 231us) server_negotiation.cc:407] Connection header received
0924 10:39:08.589254 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:08.589257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:08.589306 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:08.589388 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:08.590211 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:08.590984 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:08.592178 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:08.592522 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:08.592633 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:08.594418 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:08.594421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:08.594424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:08.594450 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:08.596041 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:08.596612 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:08.596616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:08.596618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:08.596705 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:08.597133 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:08.597136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:08.597137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:08.597396 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:08.597540 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:08.597884 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 10:39:08.597979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:39:08.603936 23833 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:08.604849 23833 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:08.611789 23833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50c69f1dc1ff44f7a682c3a8bc387cb8: Generated new TSK 0
I20260924 10:39:08.612648 23833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:08.624853 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23750
W20260924 10:39:08.629420 23735 connection.cc:535] client connection to 127.0.0.1:33585 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33585): Connection reset by peer (error 104)
2026-09-24T10:39:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:08 dist-test-slave-8wc7 krb5kdc[23849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:08 dist-test-slave-8wc7 krb5kdc[23849](info): set up 2 sockets
Sep 24 10:39:08 dist-test-slave-8wc7 krb5kdc[23849](info): commencing operation
krb5kdc: starting...
W20260924 10:39:09.657502 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:09 dist-test-slave-8wc7 krb5kdc[23849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:09.783861 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:09.896821 23862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:09.897131 23862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:09.897253 23862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:09.901348 23862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:09.901418 23862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:09.901438 23862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:09.901455 23862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:09.901472 23862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:09.907189 23862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:09.908327 23862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:09.909394 23862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:09.917922 23867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:09.917917 23870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:09.918044 23862 server_base.cc:1034] running on GCE node
W20260924 10:39:09.918258 23868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:09.918851 23862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:09.920040 23862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:09.921280 23862 hybrid_clock.cc:648] HybridClock initialized: now 1790246349921237 us; error 47 us; skew 500 ppm
Sep 24 10:39:09 dist-test-slave-8wc7 krb5kdc[23849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:09.930253 23862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:09.931324 23862 webserver.cc:460] Webserver started at http://127.0.0.1:37809/ using document root <none> and password file <none>
I20260924 10:39:09.931942 23862 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:09.932039 23862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:09.932287 23862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:09.934561 23862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "1d623f72cf1b49b58e0120cd387bec95"
format_stamp: "Formatted at 2026-09-24 10:39:09 on dist-test-slave-8wc7"
server_key: "432bf448ecf2f289b6cf14be1065cbb9"
server_key_iv: "8c517c47bc0374c8b34bc7e589253ead"
server_key_version: "encryptionkey@0"
I20260924 10:39:09.935173 23862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "1d623f72cf1b49b58e0120cd387bec95"
format_stamp: "Formatted at 2026-09-24 10:39:09 on dist-test-slave-8wc7"
server_key: "432bf448ecf2f289b6cf14be1065cbb9"
server_key_iv: "8c517c47bc0374c8b34bc7e589253ead"
server_key_version: "encryptionkey@0"
I20260924 10:39:09.939242 23862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:39:09.941923 23877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:09.943233 23862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:39:09.943388 23862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "1d623f72cf1b49b58e0120cd387bec95"
format_stamp: "Formatted at 2026-09-24 10:39:09 on dist-test-slave-8wc7"
server_key: "432bf448ecf2f289b6cf14be1065cbb9"
server_key_iv: "8c517c47bc0374c8b34bc7e589253ead"
server_key_version: "encryptionkey@0"
I20260924 10:39:09.943521 23862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:09.970366 23862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:09.974073 23862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:09.974294 23862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:09.982040 23862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36743
I20260924 10:39:09.982055 23929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36743 every 8 connection(s)
I20260924 10:39:09.983093 23862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:09.985984 23930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:09.990676 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23862
I20260924 10:39:09.990820 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:09.991127 2822 external_mini_cluster.cc:1428] Setting key 6901de62c6d8d8a39ce53e943a4fe193
I20260924 10:39:09.992537 23930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95: Bootstrap starting.
I20260924 10:39:09.995275 23930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:09.996054 23930 log.cc:824] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:09.998122 23930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95: No bootstrap required, opened a new log
I20260924 10:39:10.000802 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:09.992637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:44395 (local address 127.0.0.1:36743)
0924 10:39:09.993242 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:09.993268 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:09.993324 (+ 56us) server_negotiation.cc:407] Connection header received
0924 10:39:09.994209 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:09.994229 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:09.994616 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:09.994978 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:09.995461 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:09.996504 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:09.997477 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:09.997971 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:09.998162 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:09.999104 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:09.999110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:09.999124 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:09.999537 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:09.999592 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:09.999615 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:09.999755 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:09.999913 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:39:10.000059 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":136,"threads_started":1}
I20260924 10:39:10.001945 23930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d623f72cf1b49b58e0120cd387bec95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36743 } }
I20260924 10:39:10.002278 23930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:10.002339 23930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d623f72cf1b49b58e0120cd387bec95, State: Initialized, Role: FOLLOWER
I20260924 10:39:10.002816 23930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d623f72cf1b49b58e0120cd387bec95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36743 } }
I20260924 10:39:10.002995 23930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:10.003087 23930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:10.003214 23930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:10.004019 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:10.004465 23930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d623f72cf1b49b58e0120cd387bec95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36743 } }
I20260924 10:39:10.004833 23930 leader_election.cc:302] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d623f72cf1b49b58e0120cd387bec95; no voters:
I20260924 10:39:10.005184 23930 leader_election.cc:288] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:10.005316 23935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:10.005581 23935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [term 1 LEADER]: Becoming Leader. State: Replica: 1d623f72cf1b49b58e0120cd387bec95, State: Running, Role: LEADER
I20260924 10:39:10.005959 23935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d623f72cf1b49b58e0120cd387bec95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36743 } }
I20260924 10:39:10.006688 23930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:10.007689 23935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d623f72cf1b49b58e0120cd387bec95. Latest consensus state: current_term: 1 leader_uuid: "1d623f72cf1b49b58e0120cd387bec95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d623f72cf1b49b58e0120cd387bec95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36743 } } }
I20260924 10:39:10.007813 23935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:10.008345 23940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d623f72cf1b49b58e0120cd387bec95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d623f72cf1b49b58e0120cd387bec95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36743 } } }
I20260924 10:39:10.008659 23940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:10.011456 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:10.006585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:38827 (local address 127.0.0.1:36743)
0924 10:39:10.006711 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:10.006715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:10.006733 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:39:10.006787 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:10.006791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:10.006837 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:10.006929 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:10.007225 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:10.008075 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:10.009281 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:10.009599 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:10.009714 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:10.010681 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:10.010685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:10.010688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:10.010702 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:10.010731 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:10.010744 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:10.010820 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:10.011273 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 10:39:10.011309 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 10:39:10.012068 23955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:10.012163 23955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:10.012246 23954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:10.012967 23954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:10.018930 23954 catalog_manager.cc:1347] Generated new cluster ID: 9d598001cd0b466e86997548a65c35a8
I20260924 10:39:10.019016 23954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:10.044932 23954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:10.045964 23954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:10.052659 23954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d623f72cf1b49b58e0120cd387bec95: Generated new TSK 0
I20260924 10:39:10.053470 23954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:10.067957 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23862
W20260924 10:39:10.073319 23847 connection.cc:535] client connection to 127.0.0.1:36743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36743): Connection reset by peer (error 104)
2026-09-24T10:39:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:10 dist-test-slave-8wc7 krb5kdc[23961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:10 dist-test-slave-8wc7 krb5kdc[23961](info): set up 2 sockets
Sep 24 10:39:10 dist-test-slave-8wc7 krb5kdc[23961](info): commencing operation
krb5kdc: starting...
W20260924 10:39:11.093472 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:11 dist-test-slave-8wc7 krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:11.219859 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:11.331513 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:11.331815 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:11.331859 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:11.335891 23974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:11.335963 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:11.335983 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:11.336000 23974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:11.336016 23974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:11.341478 23974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:11.342586 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:11.343641 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:11.352381 23979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:11.352380 23982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:11.352381 23980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:11.352777 23974 server_base.cc:1034] running on GCE node
I20260924 10:39:11.353381 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:11.354528 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:11.355693 23974 hybrid_clock.cc:648] HybridClock initialized: now 1790246351355675 us; error 34 us; skew 500 ppm
Sep 24 10:39:11 dist-test-slave-8wc7 krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:11.358727 23974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:11.359701 23974 webserver.cc:460] Webserver started at http://127.0.0.1:37729/ using document root <none> and password file <none>
I20260924 10:39:11.360301 23974 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:11.360397 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:11.360642 23974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:11.362962 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "65814b91804e44b9a3ff751c9167ef64"
format_stamp: "Formatted at 2026-09-24 10:39:11 on dist-test-slave-8wc7"
server_key: "4220a2d6de86a86581c00756b66cb625"
server_key_iv: "25c9f5e57c94d6fcd82a5105d721d928"
server_key_version: "encryptionkey@0"
I20260924 10:39:11.363548 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "65814b91804e44b9a3ff751c9167ef64"
format_stamp: "Formatted at 2026-09-24 10:39:11 on dist-test-slave-8wc7"
server_key: "4220a2d6de86a86581c00756b66cb625"
server_key_iv: "25c9f5e57c94d6fcd82a5105d721d928"
server_key_version: "encryptionkey@0"
I20260924 10:39:11.367388 23974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:39:11.369930 23989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:11.371232 23974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:11.371381 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "65814b91804e44b9a3ff751c9167ef64"
format_stamp: "Formatted at 2026-09-24 10:39:11 on dist-test-slave-8wc7"
server_key: "4220a2d6de86a86581c00756b66cb625"
server_key_iv: "25c9f5e57c94d6fcd82a5105d721d928"
server_key_version: "encryptionkey@0"
I20260924 10:39:11.371502 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:11.376235 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:11.379247 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:11.379452 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:11.386647 23974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39597
I20260924 10:39:11.386662 24041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39597 every 8 connection(s)
I20260924 10:39:11.387686 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:11.390621 24042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:11.396365 24042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64: Bootstrap starting.
I20260924 10:39:11.396627 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 23974
I20260924 10:39:11.396775 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:11.397064 2822 external_mini_cluster.cc:1428] Setting key 680a88fcf4ac824fabea2d7c9c469c0f
I20260924 10:39:11.399096 24042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:11.399850 24042 log.cc:824] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:11.402017 24042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64: No bootstrap required, opened a new log
I20260924 10:39:11.402452 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:11.398426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:38203 (local address 127.0.0.1:39597)
0924 10:39:11.398892 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:11.398906 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:11.398941 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:39:11.399632 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:11.399650 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:11.399940 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:11.400325 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:11.401128 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:11.401133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:11.401154 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:11.401164 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:11.401279 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:11.401302 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:11.401410 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:11.401530 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:39:11.401688 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
I20260924 10:39:11.405330 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:11.406241 24042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65814b91804e44b9a3ff751c9167ef64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260924 10:39:11.406500 24042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:11.406562 24042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65814b91804e44b9a3ff751c9167ef64, State: Initialized, Role: FOLLOWER
I20260924 10:39:11.407037 24042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65814b91804e44b9a3ff751c9167ef64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260924 10:39:11.407192 24042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:11.407280 24042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:11.407382 24042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:11.408732 24042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65814b91804e44b9a3ff751c9167ef64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260924 10:39:11.409073 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:11.407411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:39511 (local address 127.0.0.1:39597)
0924 10:39:11.407566 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:11.407573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:11.407712 (+ 139us) server_negotiation.cc:407] Connection header received
0924 10:39:11.407808 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:11.407811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:11.407846 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:11.407930 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:11.408632 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:11.408635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:11.408638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:11.408653 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:11.408685 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:11.408701 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:11.408794 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:11.408896 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 10:39:11.408939 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:39:11.409149 24042 leader_election.cc:302] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65814b91804e44b9a3ff751c9167ef64; no voters:
I20260924 10:39:11.409540 24042 leader_election.cc:288] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:11.409619 24052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:11.409823 24052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [term 1 LEADER]: Becoming Leader. State: Replica: 65814b91804e44b9a3ff751c9167ef64, State: Running, Role: LEADER
I20260924 10:39:11.410132 24052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65814b91804e44b9a3ff751c9167ef64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } }
I20260924 10:39:11.410893 24042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:11.411904 24053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65814b91804e44b9a3ff751c9167ef64. Latest consensus state: current_term: 1 leader_uuid: "65814b91804e44b9a3ff751c9167ef64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65814b91804e44b9a3ff751c9167ef64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } } }
I20260924 10:39:11.412051 24053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:11.412325 24054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65814b91804e44b9a3ff751c9167ef64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65814b91804e44b9a3ff751c9167ef64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39597 } } }
I20260924 10:39:11.412415 24054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:11.412539 24061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:11.415653 24061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:11.421347 24061 catalog_manager.cc:1347] Generated new cluster ID: d94692f0eb9c43218ce8a822e732a5d6
I20260924 10:39:11.421437 24061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:11.454051 24061 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:11.454427 24061 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:11.466282 24061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65814b91804e44b9a3ff751c9167ef64: Generated new TSK 0
I20260924 10:39:11.467118 24061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:11.471395 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 23974
2026-09-24T10:39:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:11 dist-test-slave-8wc7 krb5kdc[24073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:11 dist-test-slave-8wc7 krb5kdc[24073](info): set up 2 sockets
Sep 24 10:39:11 dist-test-slave-8wc7 krb5kdc[24073](info): commencing operation
krb5kdc: starting...
W20260924 10:39:12.521571 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:12 dist-test-slave-8wc7 krb5kdc[24073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:12.649307 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:12.762562 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:12.762921 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:12.763011 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:12.767166 24086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:12.767287 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:12.767328 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:12.767366 24086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:12.767395 24086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:12.772948 24086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44119
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:12.774139 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:12.775230 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:12.783890 24091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:12.783959 24086 server_base.cc:1034] running on GCE node
W20260924 10:39:12.783905 24094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:12.783918 24092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:12.784847 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:12.786172 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:12.787358 24086 hybrid_clock.cc:648] HybridClock initialized: now 1790246352787334 us; error 54 us; skew 500 ppm
Sep 24 10:39:12 dist-test-slave-8wc7 krb5kdc[24073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:12.790268 24086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:12.791229 24086 webserver.cc:460] Webserver started at http://127.0.0.1:46025/ using document root <none> and password file <none>
I20260924 10:39:12.791803 24086 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:12.791865 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:12.792088 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:12.794250 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "dda8030c052f433a948061590b3b1d9b"
format_stamp: "Formatted at 2026-09-24 10:39:12 on dist-test-slave-8wc7"
server_key: "8cbdd30071180f151efbf8133bcbfd11"
server_key_iv: "f64bd664746dc299926aa2d91ec1097e"
server_key_version: "encryptionkey@0"
I20260924 10:39:12.794818 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "dda8030c052f433a948061590b3b1d9b"
format_stamp: "Formatted at 2026-09-24 10:39:12 on dist-test-slave-8wc7"
server_key: "8cbdd30071180f151efbf8133bcbfd11"
server_key_iv: "f64bd664746dc299926aa2d91ec1097e"
server_key_version: "encryptionkey@0"
I20260924 10:39:12.798566 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:39:12.800976 24101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:12.802244 24086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:12.802374 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "dda8030c052f433a948061590b3b1d9b"
format_stamp: "Formatted at 2026-09-24 10:39:12 on dist-test-slave-8wc7"
server_key: "8cbdd30071180f151efbf8133bcbfd11"
server_key_iv: "f64bd664746dc299926aa2d91ec1097e"
server_key_version: "encryptionkey@0"
I20260924 10:39:12.802485 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:12.812623 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:12.815502 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:12.815707 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:12.823027 24086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44119
I20260924 10:39:12.823048 24153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44119 every 8 connection(s)
I20260924 10:39:12.824041 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:12.825909 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24086
I20260924 10:39:12.826071 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:12.826375 2822 external_mini_cluster.cc:1428] Setting key a697f92a5b32253f34d1d23911e1d73b
I20260924 10:39:12.827325 24154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:12.833079 24154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b: Bootstrap starting.
Sep 24 10:39:12 dist-test-slave-8wc7 krb5kdc[24073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:39:12.835716 24154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:12.836608 24154 log.cc:824] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:12.838680 24154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b: No bootstrap required, opened a new log
I20260924 10:39:12.840660 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:12.827871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:60369 (local address 127.0.0.1:44119)
0924 10:39:12.828323 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:12.828339 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:12.828376 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:39:12.829005 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:12.829022 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:12.829445 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:12.829771 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:12.830570 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:12.831548 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:12.832413 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:12.832885 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:12.833055 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:12.835804 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:12.835809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:12.835843 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:12.835879 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:12.837777 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:12.838316 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:12.838321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:12.838326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:12.838432 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:12.838779 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:12.838784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:12.838786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:12.839181 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:12.839397 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:12.839710 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 10:39:12.839941 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260924 10:39:12.842731 24154 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda8030c052f433a948061590b3b1d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260924 10:39:12.842993 24154 raft_consensus.cc:374] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:12.843055 24154 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda8030c052f433a948061590b3b1d9b, State: Initialized, Role: FOLLOWER
I20260924 10:39:12.843590 24154 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda8030c052f433a948061590b3b1d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260924 10:39:12.843763 24154 raft_consensus.cc:388] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:12.843793 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:12.843828 24154 raft_consensus.cc:482] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:12.843920 24154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:12.844919 24154 raft_consensus.cc:504] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda8030c052f433a948061590b3b1d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260924 10:39:12.845347 24154 leader_election.cc:302] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dda8030c052f433a948061590b3b1d9b; no voters:
I20260924 10:39:12.845698 24154 leader_election.cc:288] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:12.845765 24163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:12.846027 24163 raft_consensus.cc:686] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [term 1 LEADER]: Becoming Leader. State: Replica: dda8030c052f433a948061590b3b1d9b, State: Running, Role: LEADER
I20260924 10:39:12.846328 24163 consensus_queue.cc:237] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda8030c052f433a948061590b3b1d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } }
I20260924 10:39:12.847255 24154 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:12.847677 24163 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dda8030c052f433a948061590b3b1d9b. Latest consensus state: current_term: 1 leader_uuid: "dda8030c052f433a948061590b3b1d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda8030c052f433a948061590b3b1d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } } }
I20260924 10:39:12.847815 24163 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:12.848086 24163 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dda8030c052f433a948061590b3b1d9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda8030c052f433a948061590b3b1d9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44119 } } }
I20260924 10:39:12.848254 24168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:12.848255 24163 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:12.851694 24168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:12.858148 24168 catalog_manager.cc:1347] Generated new cluster ID: 18a5634994724f6cae3dc2a958ce743e
I20260924 10:39:12.858503 24168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:12.860865 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:12.845904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:43585 (local address 127.0.0.1:44119)
0924 10:39:12.846030 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:12.846034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:12.849676 (+ 3642us) server_negotiation.cc:407] Connection header received
0924 10:39:12.849984 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:12.849987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:12.850039 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:12.850130 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:12.850984 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:12.851692 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:12.852973 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:12.853322 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:12.853430 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:12.857293 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:12.857297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:12.857300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:12.857328 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:12.859009 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:12.859561 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:12.859564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:12.859565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:12.859638 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:12.859967 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:12.859970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:12.859972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:12.860159 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:12.860315 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:12.860577 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 10:39:12.860678 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:39:12.885862 24168 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:12.887212 24168 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:12.899791 24168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dda8030c052f433a948061590b3b1d9b: Generated new TSK 0
I20260924 10:39:12.900727 24168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:12.906462 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24086
W20260924 10:39:12.911803 24071 connection.cc:535] client connection to 127.0.0.1:44119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44119): Connection reset by peer (error 104)
2026-09-24T10:39:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:12 dist-test-slave-8wc7 krb5kdc[24185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:12 dist-test-slave-8wc7 krb5kdc[24185](info): set up 2 sockets
Sep 24 10:39:12 dist-test-slave-8wc7 krb5kdc[24185](info): commencing operation
krb5kdc: starting...
W20260924 10:39:13.941447 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:14 dist-test-slave-8wc7 krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:14.064622 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:14.177776 24198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:14.178068 24198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:14.178153 24198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:14.182222 24198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:14.182287 24198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:14.182307 24198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:14.182374 24198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:14.182403 24198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:14.187790 24198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:14.188997 24198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:14.190135 24198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:14.199163 24203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:14.199153 24204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:14.199153 24206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:14.199203 24198 server_base.cc:1034] running on GCE node
I20260924 10:39:14.200044 24198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:14.201215 24198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:14.202411 24198 hybrid_clock.cc:648] HybridClock initialized: now 1790246354202396 us; error 27 us; skew 500 ppm
Sep 24 10:39:14 dist-test-slave-8wc7 krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:14.205112 24198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:14.206002 24198 webserver.cc:460] Webserver started at http://127.0.0.1:36763/ using document root <none> and password file <none>
I20260924 10:39:14.206568 24198 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:14.206673 24198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:14.206909 24198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:14.208963 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "a821d9aee959400fb6a057fd39c71239"
format_stamp: "Formatted at 2026-09-24 10:39:14 on dist-test-slave-8wc7"
server_key: "c1287c621f3e1fa77457bc81fab4fb7f"
server_key_iv: "07bc46c3645a40c83f7d3c96bff80788"
server_key_version: "encryptionkey@0"
I20260924 10:39:14.209566 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "a821d9aee959400fb6a057fd39c71239"
format_stamp: "Formatted at 2026-09-24 10:39:14 on dist-test-slave-8wc7"
server_key: "c1287c621f3e1fa77457bc81fab4fb7f"
server_key_iv: "07bc46c3645a40c83f7d3c96bff80788"
server_key_version: "encryptionkey@0"
I20260924 10:39:14.213376 24198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:14.215837 24213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:14.217150 24198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:14.217350 24198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "a821d9aee959400fb6a057fd39c71239"
format_stamp: "Formatted at 2026-09-24 10:39:14 on dist-test-slave-8wc7"
server_key: "c1287c621f3e1fa77457bc81fab4fb7f"
server_key_iv: "07bc46c3645a40c83f7d3c96bff80788"
server_key_version: "encryptionkey@0"
I20260924 10:39:14.217461 24198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:14.246990 24198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:14.251111 24198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:14.251322 24198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:14.258221 24198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40297
I20260924 10:39:14.258292 24265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40297 every 8 connection(s)
I20260924 10:39:14.259218 24198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:14.261302 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24198
I20260924 10:39:14.261448 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:14.261742 2822 external_mini_cluster.cc:1428] Setting key eb0256483514358d5e7d96abd09ed155
I20260924 10:39:14.262226 24266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:14.269389 24266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239: Bootstrap starting.
I20260924 10:39:14.272064 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:14.263119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:56007 (local address 127.0.0.1:40297)
0924 10:39:14.263546 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:14.263566 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:14.263609 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:39:14.264503 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:14.264534 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:14.264973 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:14.265375 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:14.265806 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:14.267246 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:14.268207 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:14.268895 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:14.269056 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:14.270043 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:14.270049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:14.270064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:14.270593 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:14.270665 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:14.270698 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:14.270836 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:14.270998 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 10:39:14.271130 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260924 10:39:14.272115 24266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:14.272933 24266 log.cc:824] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:14.275092 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:14.275168 24266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239: No bootstrap required, opened a new log
I20260924 10:39:14.279018 24266 raft_consensus.cc:348] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821d9aee959400fb6a057fd39c71239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260924 10:39:14.279281 24266 raft_consensus.cc:374] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:14.279342 24266 raft_consensus.cc:729] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a821d9aee959400fb6a057fd39c71239, State: Initialized, Role: FOLLOWER
I20260924 10:39:14.279767 24266 consensus_queue.cc:260] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821d9aee959400fb6a057fd39c71239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260924 10:39:14.279915 24266 raft_consensus.cc:388] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:14.279984 24266 raft_consensus.cc:482] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:14.280107 24266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:14.280985 24266 raft_consensus.cc:504] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821d9aee959400fb6a057fd39c71239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260924 10:39:14.281147 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:14.277076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:34315 (local address 127.0.0.1:40297)
0924 10:39:14.277212 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:14.277216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:14.277388 (+ 172us) server_negotiation.cc:407] Connection header received
0924 10:39:14.277556 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:14.277558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:14.277595 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:14.277684 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:14.277953 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:14.278455 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:14.279302 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:14.279695 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:14.279808 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:14.280695 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:14.280699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:14.280702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:14.280714 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:14.280747 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:14.280758 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:14.280853 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:14.280970 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 10:39:14.281005 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:39:14.281399 24266 leader_election.cc:302] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a821d9aee959400fb6a057fd39c71239; no voters:
I20260924 10:39:14.281723 24266 leader_election.cc:288] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:14.281864 24276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:14.282150 24276 raft_consensus.cc:686] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [term 1 LEADER]: Becoming Leader. State: Replica: a821d9aee959400fb6a057fd39c71239, State: Running, Role: LEADER
I20260924 10:39:14.282506 24276 consensus_queue.cc:237] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821d9aee959400fb6a057fd39c71239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260924 10:39:14.283013 24266 sys_catalog.cc:564] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:14.283583 24276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a821d9aee959400fb6a057fd39c71239. Latest consensus state: current_term: 1 leader_uuid: "a821d9aee959400fb6a057fd39c71239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821d9aee959400fb6a057fd39c71239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } } }
I20260924 10:39:14.283769 24276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:14.284014 24278 sys_catalog.cc:455] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a821d9aee959400fb6a057fd39c71239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a821d9aee959400fb6a057fd39c71239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } } }
I20260924 10:39:14.284137 24278 sys_catalog.cc:458] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:14.284293 24285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:14.287493 24285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:14.293251 24285 catalog_manager.cc:1347] Generated new cluster ID: 526dfa29c8004d3683a6e36bad7b4a05
I20260924 10:39:14.293316 24285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:14.302038 24285 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:14.302966 24285 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:14.310418 24285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a821d9aee959400fb6a057fd39c71239: Generated new TSK 0
I20260924 10:39:14.311082 24285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:14.318696 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24198
W20260924 10:39:14.324170 24183 connection.cc:535] client connection to 127.0.0.1:40297 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40297): Connection reset by peer (error 104)
2026-09-24T10:39:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:14 dist-test-slave-8wc7 krb5kdc[24297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:14 dist-test-slave-8wc7 krb5kdc[24297](info): set up 2 sockets
Sep 24 10:39:14 dist-test-slave-8wc7 krb5kdc[24297](info): commencing operation
krb5kdc: starting...
W20260924 10:39:15.341487 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:15 dist-test-slave-8wc7 krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:39:15.465098 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:15.577819 24310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:15.578145 24310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:15.578224 24310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:15.582343 24310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:15.582441 24310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:15.582477 24310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:15.582536 24310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:15.582576 24310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:15.587934 24310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:15.589073 24310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:15.590161 24310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:39:15.598767 24310 server_base.cc:1034] running on GCE node
W20260924 10:39:15.598825 24315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:15.598789 24316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:15.598987 24318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:15.599596 24310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:15.600723 24310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:15.601909 24310 hybrid_clock.cc:648] HybridClock initialized: now 1790246355601881 us; error 42 us; skew 500 ppm
Sep 24 10:39:15 dist-test-slave-8wc7 krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:15.608286 24310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:39:15.609297 24310 webserver.cc:460] Webserver started at http://127.0.0.1:36085/ using document root <none> and password file <none>
I20260924 10:39:15.609874 24310 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:15.609968 24310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:15.610208 24310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:15.612324 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "99c3be5de213497c8f73b6299dbca641"
format_stamp: "Formatted at 2026-09-24 10:39:15 on dist-test-slave-8wc7"
server_key: "cf4afdf853dca8142fbf48979c6f6ae7"
server_key_iv: "abaf0b8c5ebaadb31c9ed591c9336035"
server_key_version: "encryptionkey@0"
I20260924 10:39:15.612900 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "99c3be5de213497c8f73b6299dbca641"
format_stamp: "Formatted at 2026-09-24 10:39:15 on dist-test-slave-8wc7"
server_key: "cf4afdf853dca8142fbf48979c6f6ae7"
server_key_iv: "abaf0b8c5ebaadb31c9ed591c9336035"
server_key_version: "encryptionkey@0"
I20260924 10:39:15.616786 24310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:39:15.619347 24325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:15.620610 24310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:15.620769 24310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "99c3be5de213497c8f73b6299dbca641"
format_stamp: "Formatted at 2026-09-24 10:39:15 on dist-test-slave-8wc7"
server_key: "cf4afdf853dca8142fbf48979c6f6ae7"
server_key_iv: "abaf0b8c5ebaadb31c9ed591c9336035"
server_key_version: "encryptionkey@0"
I20260924 10:39:15.620888 24310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:15.625505 24310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:15.628450 24310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:15.628652 24310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:15.635650 24310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41327
I20260924 10:39:15.635675 24377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41327 every 8 connection(s)
I20260924 10:39:15.636605 24310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:15.639468 24378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:15.641733 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24310
I20260924 10:39:15.641891 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:15.642158 2822 external_mini_cluster.cc:1428] Setting key e560d7d279f6823e059562bdb64540cd
I20260924 10:39:15.645686 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641: Bootstrap starting.
I20260924 10:39:15.647404 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:15.643509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:37841 (local address 127.0.0.1:41327)
0924 10:39:15.643971 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:15.643986 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:15.644022 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:39:15.644748 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:15.644766 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:15.645054 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:15.645452 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:15.646232 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:15.646238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:15.646259 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:15.646269 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:15.646319 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:15.646339 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:15.646477 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:15.646558 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:39:15.646730 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":148,"threads_started":1}
I20260924 10:39:15.648103 24378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:15.648984 24378 log.cc:824] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:15.650346 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:15.651147 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641: No bootstrap required, opened a new log
I20260924 10:39:15.654116 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:15.652440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.20:35875 (local address 127.0.0.1:41327)
0924 10:39:15.652558 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:15.652568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:15.652703 (+ 135us) server_negotiation.cc:407] Connection header received
0924 10:39:15.652831 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:15.652833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:15.652862 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:15.652919 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:15.653708 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:15.653711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:15.653713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:15.653723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:15.653747 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:15.653756 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:15.653824 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:15.653937 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 10:39:15.653970 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:39:15.655664 24378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3be5de213497c8f73b6299dbca641" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 10:39:15.655932 24378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:15.655992 24378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c3be5de213497c8f73b6299dbca641, State: Initialized, Role: FOLLOWER
I20260924 10:39:15.656407 24378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3be5de213497c8f73b6299dbca641" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 10:39:15.656566 24378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:15.656656 24378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:15.656754 24378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:15.657765 24378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3be5de213497c8f73b6299dbca641" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 10:39:15.658113 24378 leader_election.cc:302] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99c3be5de213497c8f73b6299dbca641; no voters:
I20260924 10:39:15.658416 24378 leader_election.cc:288] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:15.658555 24388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:15.658779 24388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [term 1 LEADER]: Becoming Leader. State: Replica: 99c3be5de213497c8f73b6299dbca641, State: Running, Role: LEADER
I20260924 10:39:15.659080 24388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3be5de213497c8f73b6299dbca641" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 10:39:15.659607 24378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:15.660185 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c3be5de213497c8f73b6299dbca641. Latest consensus state: current_term: 1 leader_uuid: "99c3be5de213497c8f73b6299dbca641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3be5de213497c8f73b6299dbca641" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } } }
I20260924 10:39:15.660349 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:15.660764 24396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:15.661697 24389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99c3be5de213497c8f73b6299dbca641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3be5de213497c8f73b6299dbca641" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } } }
I20260924 10:39:15.661882 24389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:15.664494 24396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:15.669812 24396 catalog_manager.cc:1347] Generated new cluster ID: 868bf10b2c3b4af597e729c8fb902a8f
I20260924 10:39:15.669888 24396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:15.692664 24396 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:15.693027 24396 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:15.699630 24396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99c3be5de213497c8f73b6299dbca641: Generated new TSK 0
I20260924 10:39:15.700476 24396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:15.711475 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24310
2026-09-24T10:39:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:15 dist-test-slave-8wc7 krb5kdc[24409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:15 dist-test-slave-8wc7 krb5kdc[24409](info): set up 2 sockets
Sep 24 10:39:15 dist-test-slave-8wc7 krb5kdc[24409](info): commencing operation
krb5kdc: starting...
W20260924 10:39:16.737581 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:16 dist-test-slave-8wc7 krb5kdc[24409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:16.859768 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:42773
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:42773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:16.973330 24422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:16.973618 24422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:16.973661 24422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:16.978011 24422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:16.978080 24422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:16.978104 24422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:16.978122 24422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:16.978137 24422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:16.983646 24422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:42773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:16.984807 24422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:16.985999 24422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:16.994583 24427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:16.994582 24430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:16.994874 24422 server_base.cc:1034] running on GCE node
W20260924 10:39:16.994724 24428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:16.995667 24422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:16.997025 24422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:16.998222 24422 hybrid_clock.cc:648] HybridClock initialized: now 1790246356998203 us; error 35 us; skew 500 ppm
Sep 24 10:39:16 dist-test-slave-8wc7 krb5kdc[24409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246356, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:17.000991 24422 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:17.001940 24422 webserver.cc:460] Webserver started at http://127.2.193.190:44841/ using document root <none> and password file <none>
I20260924 10:39:17.002523 24422 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:17.002616 24422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:17.002853 24422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:17.004912 24422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "8349d02b328144d4ad5fe0ba79665a48"
format_stamp: "Formatted at 2026-09-24 10:39:16 on dist-test-slave-8wc7"
server_key: "2394f7b6b336212bab85d645ec10db6f"
server_key_iv: "08ba45a03be51269b47fdb6c996e6a41"
server_key_version: "encryptionkey@0"
I20260924 10:39:17.005553 24422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "8349d02b328144d4ad5fe0ba79665a48"
format_stamp: "Formatted at 2026-09-24 10:39:16 on dist-test-slave-8wc7"
server_key: "2394f7b6b336212bab85d645ec10db6f"
server_key_iv: "08ba45a03be51269b47fdb6c996e6a41"
server_key_version: "encryptionkey@0"
I20260924 10:39:17.009387 24422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:39:17.011783 24437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:17.013051 24422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:17.013252 24422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "8349d02b328144d4ad5fe0ba79665a48"
format_stamp: "Formatted at 2026-09-24 10:39:16 on dist-test-slave-8wc7"
server_key: "2394f7b6b336212bab85d645ec10db6f"
server_key_iv: "08ba45a03be51269b47fdb6c996e6a41"
server_key_version: "encryptionkey@0"
I20260924 10:39:17.013370 24422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:17.029227 24422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:17.032169 24422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:17.032383 24422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:17.039400 24422 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:42773
I20260924 10:39:17.039469 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:42773 every 8 connection(s)
I20260924 10:39:17.040410 24422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:17.043274 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:17.046419 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24422
I20260924 10:39:17.046582 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:17.046859 2822 external_mini_cluster.cc:1428] Setting key 09bedd9c991c0b0181affc6fc63af145
I20260924 10:39:17.049515 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48: Bootstrap starting.
I20260924 10:39:17.052330 24490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:17.053254 24490 log.cc:824] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:17.055531 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48: No bootstrap required, opened a new log
Sep 24 10:39:17 dist-test-slave-8wc7 krb5kdc[24409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:39:17.059259 24490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8349d02b328144d4ad5fe0ba79665a48" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42773 } }
I20260924 10:39:17.059518 24490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:17.059597 24490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8349d02b328144d4ad5fe0ba79665a48, State: Initialized, Role: FOLLOWER
I20260924 10:39:17.060099 24490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8349d02b328144d4ad5fe0ba79665a48" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42773 } }
I20260924 10:39:17.060250 24490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:17.060336 24490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:17.060434 24490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:17.061444 24490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8349d02b328144d4ad5fe0ba79665a48" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42773 } }
I20260924 10:39:17.061811 24490 leader_election.cc:302] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8349d02b328144d4ad5fe0ba79665a48; no voters:
I20260924 10:39:17.062132 24490 leader_election.cc:288] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:17.062248 24495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:17.062458 24495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [term 1 LEADER]: Becoming Leader. State: Replica: 8349d02b328144d4ad5fe0ba79665a48, State: Running, Role: LEADER
I20260924 10:39:17.062635 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:17.048358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47035 (local address 127.2.193.190:42773)
0924 10:39:17.048829 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:17.048847 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:17.048892 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:39:17.049772 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:17.049795 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:17.050241 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:17.050623 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:17.051304 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:17.052797 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:17.053751 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:17.054348 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:17.054472 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:17.057322 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:17.057327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:17.057340 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:17.057367 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:17.059171 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:17.059745 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:17.059751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:17.059755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:17.059870 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:17.060225 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:17.060230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:17.060250 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:17.060799 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:17.061034 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:17.061381 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 10:39:17.061626 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":119,"threads_started":1}
I20260924 10:39:17.062831 24495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8349d02b328144d4ad5fe0ba79665a48" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42773 } }
I20260924 10:39:17.063470 24490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:17.064744 24496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8349d02b328144d4ad5fe0ba79665a48. Latest consensus state: current_term: 1 leader_uuid: "8349d02b328144d4ad5fe0ba79665a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8349d02b328144d4ad5fe0ba79665a48" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42773 } } }
I20260924 10:39:17.064909 24496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:17.065376 24505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:17.064497 24497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8349d02b328144d4ad5fe0ba79665a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8349d02b328144d4ad5fe0ba79665a48" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42773 } } }
I20260924 10:39:17.066053 24497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:17.066428 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:17.068883 24505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:17.075037 24505 catalog_manager.cc:1347] Generated new cluster ID: f7c0faeea957490c8fb9f022366fb412
I20260924 10:39:17.075430 24505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:17.077970 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:17.068694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60595 (local address 127.2.193.190:42773)
0924 10:39:17.068791 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:17.068795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:17.069242 (+ 447us) server_negotiation.cc:407] Connection header received
0924 10:39:17.069439 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:17.069442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:17.069489 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:17.069579 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:17.070429 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:17.071056 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:17.071898 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:17.072156 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:17.072273 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:17.074152 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:17.074155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:17.074158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:17.074184 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:17.076061 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:17.076648 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:17.076651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:17.076653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:17.076740 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:17.077093 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:17.077095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:17.077097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:17.077312 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:17.077452 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:17.077732 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 10:39:17.077825 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 10:39:17.084156 24505 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:17.085063 24505 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:17.096592 24505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8349d02b328144d4ad5fe0ba79665a48: Generated new TSK 0
I20260924 10:39:17.097455 24505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:17.109447 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24422
W20260924 10:39:17.113987 24407 connection.cc:535] client connection to 127.2.193.190:42773 recv error: Network error: failed to read from TLS socket (remote: 127.2.193.190:42773): Connection reset by peer (error 104)
2026-09-24T10:39:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:17 dist-test-slave-8wc7 krb5kdc[24521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:17 dist-test-slave-8wc7 krb5kdc[24521](info): set up 2 sockets
Sep 24 10:39:17 dist-test-slave-8wc7 krb5kdc[24521](info): commencing operation
krb5kdc: starting...
W20260924 10:39:18.141572 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:18 dist-test-slave-8wc7 krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:18.272053 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36677
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:39595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:36677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:18.385568 24534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:18.385871 24534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:18.385915 24534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:18.390143 24534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:18.390218 24534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:18.390239 24534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:18.390256 24534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:18.390314 24534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:18.395768 24534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:39595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:36677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:36677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:18.396962 24534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:18.398118 24534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:39:18.407096 24534 server_base.cc:1034] running on GCE node
W20260924 10:39:18.407014 24540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:18.407035 24539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:18.407014 24542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:18.407891 24534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:18.409066 24534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:18.410238 24534 hybrid_clock.cc:648] HybridClock initialized: now 1790246358410222 us; error 30 us; skew 500 ppm
Sep 24 10:39:18 dist-test-slave-8wc7 krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:18.413036 24534 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:18.414057 24534 webserver.cc:460] Webserver started at http://127.2.193.190:36229/ using document root <none> and password file <none>
I20260924 10:39:18.414656 24534 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:18.414754 24534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:18.415012 24534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:18.417412 24534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "697b033384b34fb8bb84aa94611c7812"
format_stamp: "Formatted at 2026-09-24 10:39:18 on dist-test-slave-8wc7"
server_key: "fb1979e16f1cd71d959718c6b0f922aa"
server_key_iv: "380cd264e7d9f3d3ad23f07f9e1047c0"
server_key_version: "encryptionkey@0"
I20260924 10:39:18.418043 24534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "697b033384b34fb8bb84aa94611c7812"
format_stamp: "Formatted at 2026-09-24 10:39:18 on dist-test-slave-8wc7"
server_key: "fb1979e16f1cd71d959718c6b0f922aa"
server_key_iv: "380cd264e7d9f3d3ad23f07f9e1047c0"
server_key_version: "encryptionkey@0"
I20260924 10:39:18.421967 24534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:18.424719 24549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:18.426045 24534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:18.426208 24534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "697b033384b34fb8bb84aa94611c7812"
format_stamp: "Formatted at 2026-09-24 10:39:18 on dist-test-slave-8wc7"
server_key: "fb1979e16f1cd71d959718c6b0f922aa"
server_key_iv: "380cd264e7d9f3d3ad23f07f9e1047c0"
server_key_version: "encryptionkey@0"
I20260924 10:39:18.426337 24534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:18.447511 24534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:18.451970 24534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:18.452221 24534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:18.461658 24534 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:36677
I20260924 10:39:18.461654 24601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:36677 every 8 connection(s)
I20260924 10:39:18.462707 24534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:18.465708 24602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:18.469007 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24534
I20260924 10:39:18.469125 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:18.469450 2822 external_mini_cluster.cc:1428] Setting key d13353cb4536fd37bfbd32ec9ad30880
I20260924 10:39:18.472049 24602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812: Bootstrap starting.
I20260924 10:39:18.474869 24602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:18.475711 24602 log.cc:824] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:18.477975 24602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812: No bootstrap required, opened a new log
I20260924 10:39:18.479784 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:18.471053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57811 (local address 127.2.193.190:36677)
0924 10:39:18.471485 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:18.471508 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:18.471553 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:39:18.472392 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:18.472417 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:18.472860 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:18.473280 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:18.473725 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:18.475211 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:18.476134 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:18.476736 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:18.476897 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:18.477789 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:18.477795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:18.477812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:18.478341 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:18.478410 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:18.478440 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:18.478571 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:18.478725 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 10:39:18.478857 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":100,"threads_started":1}
I20260924 10:39:18.482517 24602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697b033384b34fb8bb84aa94611c7812" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36677 } }
I20260924 10:39:18.482853 24602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:18.482937 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:18.482949 24602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697b033384b34fb8bb84aa94611c7812, State: Initialized, Role: FOLLOWER
I20260924 10:39:18.483433 24602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697b033384b34fb8bb84aa94611c7812" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36677 } }
I20260924 10:39:18.483620 24602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:18.483690 24602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:18.483795 24602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:18.485116 24602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697b033384b34fb8bb84aa94611c7812" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36677 } }
I20260924 10:39:18.485558 24602 leader_election.cc:302] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 697b033384b34fb8bb84aa94611c7812; no voters:
I20260924 10:39:18.485956 24602 leader_election.cc:288] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:18.486018 24612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:18.486230 24612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [term 1 LEADER]: Becoming Leader. State: Replica: 697b033384b34fb8bb84aa94611c7812, State: Running, Role: LEADER
I20260924 10:39:18.486548 24612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697b033384b34fb8bb84aa94611c7812" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36677 } }
I20260924 10:39:18.487326 24602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:18.487787 24612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 697b033384b34fb8bb84aa94611c7812. Latest consensus state: current_term: 1 leader_uuid: "697b033384b34fb8bb84aa94611c7812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697b033384b34fb8bb84aa94611c7812" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36677 } } }
I20260924 10:39:18.487955 24612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:18.488215 24612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "697b033384b34fb8bb84aa94611c7812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "697b033384b34fb8bb84aa94611c7812" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 36677 } } }
I20260924 10:39:18.488375 24612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:18.488693 24617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:18.489988 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:18.485312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58973 (local address 127.2.193.190:36677)
0924 10:39:18.485431 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:18.485435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:18.485653 (+ 218us) server_negotiation.cc:407] Connection header received
0924 10:39:18.485772 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:18.485775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:18.485827 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:18.485918 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:18.486225 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:18.486879 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:18.487732 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:18.488055 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:18.488166 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:18.489263 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:18.489267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:18.489270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:18.489283 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:18.489308 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:18.489335 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:18.489437 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:18.489815 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 10:39:18.489848 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:39:18.492516 24617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:18.498194 24617 catalog_manager.cc:1347] Generated new cluster ID: 476d7e6a37714480b7f0b1d6eed2ae61
I20260924 10:39:18.498270 24617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:18.516954 24617 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:18.518185 24617 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:18.526757 24617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 697b033384b34fb8bb84aa94611c7812: Generated new TSK 0
I20260924 10:39:18.527467 24617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:18.538976 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24534
W20260924 10:39:18.544159 24519 connection.cc:535] client connection to 127.2.193.190:36677 recv error: Network error: failed to read from TLS socket (remote: 127.2.193.190:36677): Connection reset by peer (error 104)
2026-09-24T10:39:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:18 dist-test-slave-8wc7 krb5kdc[24633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:18 dist-test-slave-8wc7 krb5kdc[24633](info): set up 2 sockets
Sep 24 10:39:18 dist-test-slave-8wc7 krb5kdc[24633](info): commencing operation
krb5kdc: starting...
W20260924 10:39:19.577636 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:19 dist-test-slave-8wc7 krb5kdc[24633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:19.701506 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46645
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:46645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:19.814908 24646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:19.815237 24646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:19.815317 24646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:19.819432 24646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:19.819536 24646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:19.819576 24646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:19.819636 24646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:19.819675 24646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:19.825055 24646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:46645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:19.826241 24646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:19.827391 24646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:19.836302 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:19.836262 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:19.836304 24646 server_base.cc:1034] running on GCE node
W20260924 10:39:19.836452 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:19.837224 24646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:19.838346 24646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:19.839535 24646 hybrid_clock.cc:648] HybridClock initialized: now 1790246359839514 us; error 36 us; skew 500 ppm
Sep 24 10:39:19 dist-test-slave-8wc7 krb5kdc[24633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:19.842244 24646 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:19.843096 24646 webserver.cc:460] Webserver started at http://127.2.193.190:32789/ using document root <none> and password file <none>
I20260924 10:39:19.843657 24646 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:19.843744 24646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:19.843976 24646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:19.846098 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "4db5b54af17847a18d6605c623082981"
format_stamp: "Formatted at 2026-09-24 10:39:19 on dist-test-slave-8wc7"
server_key: "219429622565e24170e90e662bdce072"
server_key_iv: "7a25ed5297827ad735b7e0307697362f"
server_key_version: "encryptionkey@0"
I20260924 10:39:19.846663 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "4db5b54af17847a18d6605c623082981"
format_stamp: "Formatted at 2026-09-24 10:39:19 on dist-test-slave-8wc7"
server_key: "219429622565e24170e90e662bdce072"
server_key_iv: "7a25ed5297827ad735b7e0307697362f"
server_key_version: "encryptionkey@0"
I20260924 10:39:19.850332 24646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:39:19.852728 24661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:19.853927 24646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:19.854075 24646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "4db5b54af17847a18d6605c623082981"
format_stamp: "Formatted at 2026-09-24 10:39:19 on dist-test-slave-8wc7"
server_key: "219429622565e24170e90e662bdce072"
server_key_iv: "7a25ed5297827ad735b7e0307697362f"
server_key_version: "encryptionkey@0"
I20260924 10:39:19.854198 24646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:19.858741 24646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:19.861420 24646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:19.861610 24646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:19.868505 24646 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:46645
I20260924 10:39:19.868558 24713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:46645 every 8 connection(s)
I20260924 10:39:19.869458 24646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:19.872071 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:19.877668 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981: Bootstrap starting.
I20260924 10:39:19.877902 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24646
I20260924 10:39:19.878033 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:19.878319 2822 external_mini_cluster.cc:1428] Setting key 0bbe03480f4fc86b5ac3244c01f6ca58
I20260924 10:39:19.880265 24714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:19.881004 24714 log.cc:824] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:19.883046 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981: No bootstrap required, opened a new log
I20260924 10:39:19.883570 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:19.879624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48201 (local address 127.2.193.190:46645)
0924 10:39:19.880040 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:19.880054 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:19.880087 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:39:19.880776 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:19.880795 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:19.881150 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:19.881540 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:19.882321 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:19.882325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:19.882336 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:19.882346 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:19.882395 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:19.882414 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:19.882537 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:19.882642 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 10:39:19.882792 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
I20260924 10:39:19.886528 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:19.887229 24714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5b54af17847a18d6605c623082981" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46645 } }
I20260924 10:39:19.887454 24714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:19.887537 24714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db5b54af17847a18d6605c623082981, State: Initialized, Role: FOLLOWER
I20260924 10:39:19.888005 24714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5b54af17847a18d6605c623082981" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46645 } }
I20260924 10:39:19.888129 24714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:19.888216 24714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:19.888334 24714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:19.889573 24714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5b54af17847a18d6605c623082981" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46645 } }
I20260924 10:39:19.889971 24714 leader_election.cc:302] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4db5b54af17847a18d6605c623082981; no voters:
I20260924 10:39:19.890316 24714 leader_election.cc:288] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:19.890440 24724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:19.890478 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:19.888642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33559 (local address 127.2.193.190:46645)
0924 10:39:19.888765 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:19.888769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:19.888936 (+ 167us) server_negotiation.cc:407] Connection header received
0924 10:39:19.889030 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:19.889033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:19.889067 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:19.889152 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:19.889980 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:19.889986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:19.889992 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:19.890005 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:19.890033 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:19.890046 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:19.890141 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:19.890258 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 10:39:19.890301 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:39:19.890679 24724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [term 1 LEADER]: Becoming Leader. State: Replica: 4db5b54af17847a18d6605c623082981, State: Running, Role: LEADER
I20260924 10:39:19.891002 24724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5b54af17847a18d6605c623082981" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46645 } }
I20260924 10:39:19.891748 24714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:19.892174 24724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db5b54af17847a18d6605c623082981. Latest consensus state: current_term: 1 leader_uuid: "4db5b54af17847a18d6605c623082981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5b54af17847a18d6605c623082981" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46645 } } }
I20260924 10:39:19.892344 24724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:19.892642 24725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4db5b54af17847a18d6605c623082981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5b54af17847a18d6605c623082981" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 46645 } } }
I20260924 10:39:19.892761 24725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:19.893901 24729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:19.896572 24729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:19.902251 24729 catalog_manager.cc:1347] Generated new cluster ID: 5c33e1f428ea43cbb0c0ba9fdc5295f2
I20260924 10:39:19.902349 24729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:19.922948 24729 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:19.923328 24729 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:19.930751 24729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4db5b54af17847a18d6605c623082981: Generated new TSK 0
I20260924 10:39:19.931488 24729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:19.939330 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24646
2026-09-24T10:39:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:19 dist-test-slave-8wc7 krb5kdc[24745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:19 dist-test-slave-8wc7 krb5kdc[24745](info): set up 2 sockets
Sep 24 10:39:19 dist-test-slave-8wc7 krb5kdc[24745](info): commencing operation
krb5kdc: starting...
W20260924 10:39:20.965456 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:21 dist-test-slave-8wc7 krb5kdc[24745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:21.090157 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:35157
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:35157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:21.204499 24758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:21.204856 24758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:21.204937 24758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:21.209087 24758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:21.209227 24758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:21.209281 24758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:21.209318 24758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:21.209350 24758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:21.214807 24758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:35157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:21.215969 24758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:21.217061 24758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:21.225744 24766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:21.225723 24763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:21.225764 24764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:21.226142 24758 server_base.cc:1034] running on GCE node
I20260924 10:39:21.226837 24758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:21.228029 24758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:21.229246 24758 hybrid_clock.cc:648] HybridClock initialized: now 1790246361229228 us; error 32 us; skew 500 ppm
Sep 24 10:39:21 dist-test-slave-8wc7 krb5kdc[24745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246361, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:21.235751 24758 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:21.236860 24758 webserver.cc:460] Webserver started at http://127.2.193.190:42907/ using document root <none> and password file <none>
I20260924 10:39:21.237491 24758 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:21.237558 24758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:21.237735 24758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:21.239807 24758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "ddf4c06f3bef47488a113529a9c9920c"
format_stamp: "Formatted at 2026-09-24 10:39:21 on dist-test-slave-8wc7"
server_key: "7dc34e5ba755ebcb437e9034441691bd"
server_key_iv: "7160ada94d491d7b9e8c217503f1e7bd"
server_key_version: "encryptionkey@0"
I20260924 10:39:21.240334 24758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "ddf4c06f3bef47488a113529a9c9920c"
format_stamp: "Formatted at 2026-09-24 10:39:21 on dist-test-slave-8wc7"
server_key: "7dc34e5ba755ebcb437e9034441691bd"
server_key_iv: "7160ada94d491d7b9e8c217503f1e7bd"
server_key_version: "encryptionkey@0"
I20260924 10:39:21.244138 24758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 10:39:21.246631 24773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:21.247931 24758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:21.248083 24758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "ddf4c06f3bef47488a113529a9c9920c"
format_stamp: "Formatted at 2026-09-24 10:39:21 on dist-test-slave-8wc7"
server_key: "7dc34e5ba755ebcb437e9034441691bd"
server_key_iv: "7160ada94d491d7b9e8c217503f1e7bd"
server_key_version: "encryptionkey@0"
I20260924 10:39:21.248217 24758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:21.263625 24758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:21.267294 24758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:21.267539 24758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:21.275097 24758 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:35157
I20260924 10:39:21.275125 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:35157 every 8 connection(s)
I20260924 10:39:21.276140 24758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:21.276930 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24758
I20260924 10:39:21.277067 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:21.277400 2822 external_mini_cluster.cc:1428] Setting key 57e964718d7fc1e16954ba1e6e3cbb97
I20260924 10:39:21.279606 24826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:21.285744 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c: Bootstrap starting.
Sep 24 10:39:21 dist-test-slave-8wc7 krb5kdc[24745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:39:21.288309 24826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:21.289059 24826 log.cc:824] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:21.291179 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c: No bootstrap required, opened a new log
I20260924 10:39:21.292245 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:21.278951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44575 (local address 127.2.193.190:35157)
0924 10:39:21.279403 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:21.279418 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:21.279456 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:39:21.280174 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:21.280192 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:21.280554 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:21.280909 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:21.281764 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:21.282791 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:21.283683 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:21.284174 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:21.284360 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:21.287057 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:21.287062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:21.287076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:21.287114 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:21.289009 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:21.289626 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:21.289631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:21.289635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:21.289747 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:21.290119 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:21.290123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:21.290125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:21.290566 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:21.290786 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:21.291121 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 10:39:21.291467 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":145,"threads_started":1}
I20260924 10:39:21.295387 24826 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf4c06f3bef47488a113529a9c9920c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35157 } }
I20260924 10:39:21.295639 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:21.295692 24826 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:21.295781 24826 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf4c06f3bef47488a113529a9c9920c, State: Initialized, Role: FOLLOWER
I20260924 10:39:21.296221 24826 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf4c06f3bef47488a113529a9c9920c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35157 } }
I20260924 10:39:21.296342 24826 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:21.296425 24826 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:21.296545 24826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:21.297675 24826 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf4c06f3bef47488a113529a9c9920c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35157 } }
I20260924 10:39:21.298084 24826 leader_election.cc:302] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf4c06f3bef47488a113529a9c9920c; no voters:
I20260924 10:39:21.298453 24826 leader_election.cc:288] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:21.298525 24836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:21.298730 24836 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [term 1 LEADER]: Becoming Leader. State: Replica: ddf4c06f3bef47488a113529a9c9920c, State: Running, Role: LEADER
I20260924 10:39:21.299041 24836 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf4c06f3bef47488a113529a9c9920c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35157 } }
I20260924 10:39:21.299824 24826 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:21.300230 24836 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf4c06f3bef47488a113529a9c9920c. Latest consensus state: current_term: 1 leader_uuid: "ddf4c06f3bef47488a113529a9c9920c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf4c06f3bef47488a113529a9c9920c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35157 } } }
I20260924 10:39:21.300403 24836 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:21.300643 24836 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf4c06f3bef47488a113529a9c9920c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf4c06f3bef47488a113529a9c9920c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 35157 } } }
I20260924 10:39:21.300797 24836 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:21.301127 24841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:21.305406 24841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:21.307019 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:21.297979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57799 (local address 127.2.193.190:35157)
0924 10:39:21.298124 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:21.298130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:21.298371 (+ 241us) server_negotiation.cc:407] Connection header received
0924 10:39:21.298522 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:21.298525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:21.298576 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:21.298657 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:21.299412 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:21.300024 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:21.301165 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:21.301522 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:21.301597 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:21.303427 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:21.303431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:21.303434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:21.303459 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:21.305103 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:21.305675 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:21.305680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:21.305682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:21.305772 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:21.306143 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:21.306146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:21.306148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:21.306341 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:21.306485 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:21.306725 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 10:39:21.306846 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:39:21.311944 24841 catalog_manager.cc:1347] Generated new cluster ID: a7b30cbd2f94477c93f676723146b983
I20260924 10:39:21.312026 24841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:21.321317 24841 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:21.322278 24841 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:21.328747 24841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf4c06f3bef47488a113529a9c9920c: Generated new TSK 0
I20260924 10:39:21.329757 24841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:21.342029 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24758
2026-09-24T10:39:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:21 dist-test-slave-8wc7 krb5kdc[24857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:21 dist-test-slave-8wc7 krb5kdc[24857](info): set up 2 sockets
Sep 24 10:39:21 dist-test-slave-8wc7 krb5kdc[24857](info): commencing operation
krb5kdc: starting...
W20260924 10:39:22.393509 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:22 dist-test-slave-8wc7 krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:22.520218 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:32897
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:32897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:22.633136 24870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:22.633495 24870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:22.633540 24870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:22.637758 24870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:22.637828 24870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:22.637847 24870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:22.637866 24870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:22.637880 24870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:22.643637 24870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:32897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:32897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:22.644804 24870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:22.645959 24870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:22.655194 24878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:22.655174 24876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:22.655174 24875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:22.655226 24870 server_base.cc:1034] running on GCE node
I20260924 10:39:22.656162 24870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:22.657444 24870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:22.658670 24870 hybrid_clock.cc:648] HybridClock initialized: now 1790246362658654 us; error 29 us; skew 500 ppm
Sep 24 10:39:22 dist-test-slave-8wc7 krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246362, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:22.661722 24870 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:22.662667 24870 webserver.cc:460] Webserver started at http://127.2.193.190:40245/ using document root <none> and password file <none>
I20260924 10:39:22.663223 24870 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:22.663285 24870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:22.663456 24870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:22.665679 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "70f056176f384487a5b6971bab74b79f"
format_stamp: "Formatted at 2026-09-24 10:39:22 on dist-test-slave-8wc7"
server_key: "ebf391ec6cb2127a19a4e1b6900f0967"
server_key_iv: "c00ce10f69a58de3041aba72610bbd4e"
server_key_version: "encryptionkey@0"
I20260924 10:39:22.666285 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "70f056176f384487a5b6971bab74b79f"
format_stamp: "Formatted at 2026-09-24 10:39:22 on dist-test-slave-8wc7"
server_key: "ebf391ec6cb2127a19a4e1b6900f0967"
server_key_iv: "c00ce10f69a58de3041aba72610bbd4e"
server_key_version: "encryptionkey@0"
I20260924 10:39:22.670281 24870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:39:22.672866 24885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:22.674242 24870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:22.674356 24870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "70f056176f384487a5b6971bab74b79f"
format_stamp: "Formatted at 2026-09-24 10:39:22 on dist-test-slave-8wc7"
server_key: "ebf391ec6cb2127a19a4e1b6900f0967"
server_key_iv: "c00ce10f69a58de3041aba72610bbd4e"
server_key_version: "encryptionkey@0"
I20260924 10:39:22.674453 24870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:22.702854 24870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:22.705943 24870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:22.706130 24870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:22.713443 24870 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:32897
I20260924 10:39:22.713459 24937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:32897 every 8 connection(s)
I20260924 10:39:22.714453 24870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:22.716878 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24870
I20260924 10:39:22.716982 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:22.717344 2822 external_mini_cluster.cc:1428] Setting key c1d9bbc646983850338ecb9cba25234d
I20260924 10:39:22.717496 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:22.724071 24938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f: Bootstrap starting.
I20260924 10:39:22.726770 24938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:22.726922 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:22.718877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40707 (local address 127.2.193.190:32897)
0924 10:39:22.719346 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:22.719362 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:22.719400 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:39:22.720274 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:22.720293 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:22.720651 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:22.721006 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:22.721487 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:22.722486 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:22.723392 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:22.723886 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:22.724072 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:22.725023 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:22.725027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:22.725040 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:22.725527 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:22.725587 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:22.725611 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:22.725729 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:22.725891 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 10:39:22.726133 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":158,"threads_started":1}
I20260924 10:39:22.727665 24938 log.cc:824] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:22.729964 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:22.730002 24938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f: No bootstrap required, opened a new log
I20260924 10:39:22.734627 24938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f056176f384487a5b6971bab74b79f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 32897 } }
I20260924 10:39:22.734935 24938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:22.734992 24938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f056176f384487a5b6971bab74b79f, State: Initialized, Role: FOLLOWER
I20260924 10:39:22.735524 24938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f056176f384487a5b6971bab74b79f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 32897 } }
I20260924 10:39:22.735662 24938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:22.735750 24938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:22.735843 24938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:22.736725 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:22.732157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49299 (local address 127.2.193.190:32897)
0924 10:39:22.732276 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:22.732280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:22.732541 (+ 261us) server_negotiation.cc:407] Connection header received
0924 10:39:22.732730 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:22.732733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:22.732791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:22.732883 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:22.733202 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:22.734016 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:22.734907 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:22.735280 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:22.735385 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:22.736294 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:22.736297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:22.736300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:22.736314 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:22.736343 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:22.736356 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:22.736443 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:22.736524 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:39:22.736581 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:39:22.737066 24938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f056176f384487a5b6971bab74b79f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 32897 } }
I20260924 10:39:22.737469 24938 leader_election.cc:302] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f056176f384487a5b6971bab74b79f; no voters:
I20260924 10:39:22.737840 24938 leader_election.cc:288] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:22.737919 24948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:22.738165 24948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [term 1 LEADER]: Becoming Leader. State: Replica: 70f056176f384487a5b6971bab74b79f, State: Running, Role: LEADER
I20260924 10:39:22.738488 24948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f056176f384487a5b6971bab74b79f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 32897 } }
I20260924 10:39:22.739194 24938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:22.740255 24950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f056176f384487a5b6971bab74b79f. Latest consensus state: current_term: 1 leader_uuid: "70f056176f384487a5b6971bab74b79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f056176f384487a5b6971bab74b79f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 32897 } } }
I20260924 10:39:22.740291 24949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f056176f384487a5b6971bab74b79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f056176f384487a5b6971bab74b79f" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 32897 } } }
I20260924 10:39:22.740397 24950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:22.740396 24949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:22.740773 24957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:22.744346 24957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:22.750936 24957 catalog_manager.cc:1347] Generated new cluster ID: 25ccff7b7a0d44b09c3bcb538df57a5f
I20260924 10:39:22.751029 24957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:22.764506 24957 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:22.765931 24957 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:22.776325 24957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f056176f384487a5b6971bab74b79f: Generated new TSK 0
I20260924 10:39:22.777316 24957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:22.782567 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24870
2026-09-24T10:39:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:22 dist-test-slave-8wc7 krb5kdc[24969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:22 dist-test-slave-8wc7 krb5kdc[24969](info): set up 2 sockets
Sep 24 10:39:22 dist-test-slave-8wc7 krb5kdc[24969](info): commencing operation
krb5kdc: starting...
W20260924 10:39:23.817523 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:23 dist-test-slave-8wc7 krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:23.942124 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:45073
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:45073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:24.056612 24982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:24.056917 24982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:24.057008 24982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:24.061254 24982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:24.061326 24982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:24.061345 24982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:24.061362 24982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:24.061424 24982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:24.066852 24982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:45073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:45073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:24.068099 24982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:24.069212 24982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:24.078002 24990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:24.078002 24988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:24.078007 24987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:24.078322 24982 server_base.cc:1034] running on GCE node
I20260924 10:39:24.078929 24982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:24.080022 24982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:24.081251 24982 hybrid_clock.cc:648] HybridClock initialized: now 1790246364081228 us; error 54 us; skew 500 ppm
Sep 24 10:39:24 dist-test-slave-8wc7 krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246364, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:24.084105 24982 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:24.085040 24982 webserver.cc:460] Webserver started at http://127.2.193.190:34217/ using document root <none> and password file <none>
I20260924 10:39:24.085644 24982 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:24.085711 24982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:24.085879 24982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:24.087945 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "1c49cfbe7b074de68ee11df8f705c33c"
format_stamp: "Formatted at 2026-09-24 10:39:24 on dist-test-slave-8wc7"
server_key: "d22a175c80feef39cb41d9f6de3eec5c"
server_key_iv: "773060018f3843a3f94c9841c30e53a6"
server_key_version: "encryptionkey@0"
I20260924 10:39:24.088529 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "1c49cfbe7b074de68ee11df8f705c33c"
format_stamp: "Formatted at 2026-09-24 10:39:24 on dist-test-slave-8wc7"
server_key: "d22a175c80feef39cb41d9f6de3eec5c"
server_key_iv: "773060018f3843a3f94c9841c30e53a6"
server_key_version: "encryptionkey@0"
I20260924 10:39:24.092331 24982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:39:24.094830 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:24.096088 24982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:24.096223 24982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "1c49cfbe7b074de68ee11df8f705c33c"
format_stamp: "Formatted at 2026-09-24 10:39:24 on dist-test-slave-8wc7"
server_key: "d22a175c80feef39cb41d9f6de3eec5c"
server_key_iv: "773060018f3843a3f94c9841c30e53a6"
server_key_version: "encryptionkey@0"
I20260924 10:39:24.096333 24982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:24.101241 24982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:24.104593 24982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:24.104794 24982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:24.112154 24982 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:45073
I20260924 10:39:24.112164 25049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:45073 every 8 connection(s)
I20260924 10:39:24.113152 24982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:24.116030 25050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:24.118975 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 24982
I20260924 10:39:24.119117 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:24.119376 2822 external_mini_cluster.cc:1428] Setting key f8003d76aad4c513e16bf3dcf414c676
I20260924 10:39:24.122224 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c: Bootstrap starting.
I20260924 10:39:24.124891 25050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:24.124917 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:24.120859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57817 (local address 127.2.193.190:45073)
0924 10:39:24.121319 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:24.121337 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:24.121373 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:39:24.122020 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:24.122037 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:24.122341 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:24.122686 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:24.123592 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:24.123596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:24.123608 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:24.123619 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:24.123673 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:24.123694 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:24.123805 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:24.123920 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 10:39:24.124094 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":146,"threads_started":1}
I20260924 10:39:24.125921 25050 log.cc:824] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:24.128082 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:24.128175 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c: No bootstrap required, opened a new log
I20260924 10:39:24.132081 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:24.130423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39105 (local address 127.2.193.190:45073)
0924 10:39:24.130564 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:24.130569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:24.130743 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:39:24.130866 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:24.130868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:24.130899 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:24.130975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:24.131672 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:24.131674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:24.131676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:24.131686 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:24.131711 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:24.131721 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:24.131789 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:24.131910 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 10:39:24.131944 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:39:24.133239 25050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c49cfbe7b074de68ee11df8f705c33c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45073 } }
I20260924 10:39:24.133504 25050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:24.133565 25050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c49cfbe7b074de68ee11df8f705c33c, State: Initialized, Role: FOLLOWER
I20260924 10:39:24.133991 25050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c49cfbe7b074de68ee11df8f705c33c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45073 } }
I20260924 10:39:24.134145 25050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:24.134228 25050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:24.134328 25050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:24.135388 25050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c49cfbe7b074de68ee11df8f705c33c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45073 } }
I20260924 10:39:24.135743 25050 leader_election.cc:302] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c49cfbe7b074de68ee11df8f705c33c; no voters:
I20260924 10:39:24.136092 25050 leader_election.cc:288] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:24.136195 25060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:24.136407 25060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [term 1 LEADER]: Becoming Leader. State: Replica: 1c49cfbe7b074de68ee11df8f705c33c, State: Running, Role: LEADER
I20260924 10:39:24.136816 25060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c49cfbe7b074de68ee11df8f705c33c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45073 } }
I20260924 10:39:24.137555 25050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:24.138619 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c49cfbe7b074de68ee11df8f705c33c. Latest consensus state: current_term: 1 leader_uuid: "1c49cfbe7b074de68ee11df8f705c33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c49cfbe7b074de68ee11df8f705c33c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45073 } } }
I20260924 10:39:24.138758 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:24.139071 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c49cfbe7b074de68ee11df8f705c33c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c49cfbe7b074de68ee11df8f705c33c" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 45073 } } }
I20260924 10:39:24.139204 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:24.139132 25069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:24.142769 25069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:24.148592 25069 catalog_manager.cc:1347] Generated new cluster ID: 39e5c95cff4b4878af035e881e5f7a4d
I20260924 10:39:24.148676 25069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:24.176535 25069 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:24.176874 25069 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:24.185045 25069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c49cfbe7b074de68ee11df8f705c33c: Generated new TSK 0
I20260924 10:39:24.185951 25069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:24.198133 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 24982
2026-09-24T10:39:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1414 ms)
[----------] 18 tests from AuthTokenIssuingTest (25505 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:24 dist-test-slave-8wc7 krb5kdc[25081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:24 dist-test-slave-8wc7 krb5kdc[25081](info): set up 2 sockets
Sep 24 10:39:24 dist-test-slave-8wc7 krb5kdc[25081](info): commencing operation
krb5kdc: starting...
W20260924 10:39:25.233570 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:25.359536 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:37411
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:37411,127.2.193.189:38791,127.2.193.188:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:25.472728 25094 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:39:25.473042 25094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:25.473066 25094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:25.473157 25094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:25.477304 25094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:25.477372 25094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:25.477391 25094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:25.477442 25094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:25.477478 25094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:25.482892 25094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:37411,127.2.193.189:38791,127.2.193.188:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:37411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:25.484062 25094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:25.485229 25094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:25.493957 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:25.493959 25099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:25.493957 25100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:25.494166 25094 server_base.cc:1034] running on GCE node
I20260924 10:39:25.494863 25094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:25.496045 25094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:25.497257 25094 hybrid_clock.cc:648] HybridClock initialized: now 1790246365497237 us; error 33 us; skew 500 ppm
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:25.500092 25094 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:25.500993 25094 webserver.cc:460] Webserver started at http://127.2.193.190:41469/ using document root <none> and password file <none>
I20260924 10:39:25.501618 25094 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:25.501709 25094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:25.501942 25094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:25.504011 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "dc9e39e6e3424f2a9abf793cce305c7d"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "7eb8a00f87672742a98dd1fb30a836bf"
server_key_iv: "43cbfe0b8dcb325830adf9b377de2085"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.504598 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "dc9e39e6e3424f2a9abf793cce305c7d"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "7eb8a00f87672742a98dd1fb30a836bf"
server_key_iv: "43cbfe0b8dcb325830adf9b377de2085"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.508500 25094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:39:25.511148 25109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:25.512426 25094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:39:25.512605 25094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "dc9e39e6e3424f2a9abf793cce305c7d"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "7eb8a00f87672742a98dd1fb30a836bf"
server_key_iv: "43cbfe0b8dcb325830adf9b377de2085"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.512737 25094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:25.536902 25094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:25.540676 25094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:25.540907 25094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:25.548432 25094 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:37411
I20260924 10:39:25.548758 25161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:37411 every 8 connection(s)
I20260924 10:39:25.549576 25094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:25.552484 25162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:25.556216 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 25094
I20260924 10:39:25.556367 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:25.556654 2822 external_mini_cluster.cc:1428] Setting key 54928a25ad4d0d6883a7fbd11a821c95
I20260924 10:39:25.556690 25162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.193.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.189@KRBTEST.COM" created.
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:39:25.574561 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.559600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.2.193.190:37411)
0924 10:39:25.560179 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:25.560194 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:25.560303 (+ 109us) server_negotiation.cc:407] Connection header received
0924 10:39:25.561494 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:25.561500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:25.561775 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:25.562001 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:25.562895 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.564401 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.565519 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.566128 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.566257 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.569421 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:25.569431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:25.569450 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:25.569483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:25.571260 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.571869 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.571874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.571875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.571983 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.572343 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.572345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.572346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.572840 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:25.572948 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:25.573459 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 10:39:25.573688 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":445,"spinlock_wait_cycles":29056,"thread_start_us":174,"threads_started":1}
I20260924 10:39:25.574569 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.559602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37411 (local address 127.0.0.1:35214)
0924 10:39:25.560115 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.560170 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:25.561061 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:25.561998 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:25.562003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:25.562345 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:25.562728 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.562742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.564541 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.564544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.565399 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.565409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.566273 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.566276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.566368 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.566981 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:25.566997 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:25.569264 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:25.571369 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.571382 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.571396 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.571748 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.572075 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.572081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.572086 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.572245 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.572969 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:25.572977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:25.573154 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 10:39:25.573382 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 10:39:25.573776 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":87,"thread_start_us":162,"threads_started":1}
I20260924 10:39:25.577479 25162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:25.578285 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.577964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38484)
0924 10:39:25.578143 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.578222 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 10:39:25.578531 25113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
I20260924 10:39:25.579344 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.579084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38488)
0924 10:39:25.579206 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.579267 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 10:39:25.579493 25162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:38791: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab.
Entry for principal kudu/127.2.193.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab.
I20260924 10:39:25.585891 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:38791
--webserver_interface=127.2.193.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab
--principal=kudu/127.2.193.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:37411,127.2.193.189:38791,127.2.193.188:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:39:25.628762 25162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } attempt: 1
I20260924 10:39:25.629681 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.629373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38496)
0924 10:39:25.629524 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.629604 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:39:25.630491 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.630279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38510)
0924 10:39:25.630385 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.630432 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:39:25.630692 25162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:38791: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
I20260924 10:39:25.698901 25162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } attempt: 2
W20260924 10:39:25.699509 25169 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260924 10:39:25.699817 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.699509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38518)
0924 10:39:25.699670 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.699737 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 10:39:25.699877 25169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:25.699952 25169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:25.700013 25169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:39:25.700670 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.700456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38522)
0924 10:39:25.700559 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.700609 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:39:25.700892 25162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:38791: Network error: Client connection negotiation failed: client connection to 127.2.193.189:38791: connect: Connection refused (error 111)
W20260924 10:39:25.704527 25169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:25.704589 25169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:25.704610 25169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:25.704629 25169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:25.704694 25169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:25.710268 25169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:37411,127.2.193.189:38791,127.2.193.188:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab
--principal=kudu/127.2.193.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.193.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:25.711390 25169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:25.712474 25169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:25.721408 25174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:25.721417 25175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:25.721473 25177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:25.721428 25169 server_base.cc:1034] running on GCE node
I20260924 10:39:25.722263 25169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:25.723453 25169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:25.724629 25169 hybrid_clock.cc:648] HybridClock initialized: now 1790246365724608 us; error 32 us; skew 500 ppm
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:25.727622 25169 init.cc:377] Logged in from keytab as kudu/127.2.193.189@KRBTEST.COM (short username kudu)
I20260924 10:39:25.728649 25169 webserver.cc:460] Webserver started at http://127.2.193.189:41085/ using document root <none> and password file <none>
I20260924 10:39:25.729311 25169 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:25.729408 25169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:25.729667 25169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:25.731892 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data/instance:
uuid: "9af2a5b3d414484f876fdead7fcc5dd7"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "4c583387bae7b523084e9512ad749295"
server_key_iv: "40ab210740dd1bec67b7e1834c7ff74a"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.732532 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/wal/instance:
uuid: "9af2a5b3d414484f876fdead7fcc5dd7"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "4c583387bae7b523084e9512ad749295"
server_key_iv: "40ab210740dd1bec67b7e1834c7ff74a"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.736686 25169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260924 10:39:25.739328 25184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:25.740834 25169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:39:25.740993 25169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/wal
uuid: "9af2a5b3d414484f876fdead7fcc5dd7"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "4c583387bae7b523084e9512ad749295"
server_key_iv: "40ab210740dd1bec67b7e1834c7ff74a"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.741124 25169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:25.750917 25169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:25.754046 25169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:25.754283 25169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:25.761492 25169 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.189:38791
I20260924 10:39:25.761773 25236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.189:38791 every 8 connection(s)
I20260924 10:39:25.762645 25169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
I20260924 10:39:25.765671 25237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:25.769582 25237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:25.772343 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 25169
I20260924 10:39:25.772496 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-1/wal/instance
I20260924 10:39:25.772835 2822 external_mini_cluster.cc:1428] Setting key 667219ad90cd9f092264bf38875eb8bf
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.193.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.188@KRBTEST.COM" created.
I20260924 10:39:25.784039 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.771550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.2.193.190:37411)
0924 10:39:25.771715 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:25.771719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:25.772671 (+ 952us) server_negotiation.cc:407] Connection header received
0924 10:39:25.773777 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:25.773784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:25.773851 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:25.773962 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:25.775149 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.775853 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.777018 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.777348 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.777468 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.780090 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:25.780099 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:25.780101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:25.780134 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:25.781709 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.782290 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.782293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.782294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.782382 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.782726 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.782729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.782730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.782915 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:25.782992 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:25.783606 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 10:39:25.783773 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 10:39:25.784891 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.771947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37411 (local address 127.0.0.1:35224)
0924 10:39:25.772484 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.772540 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:25.773545 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:25.774124 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:25.774131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:25.774478 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:25.774986 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.774999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.775974 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.775976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.776873 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.776882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.777516 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.777520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.777717 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.778385 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:25.778402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:25.779897 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:25.781845 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.781851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.781868 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.782185 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.782473 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.782475 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.782477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.782641 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.783002 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:25.783007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:25.783339 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 10:39:25.783511 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 10:39:25.783874 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":173,"threads_started":1}
I20260924 10:39:25.786505 25237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:25.792100 25162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } attempt: 3
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab.
Entry for principal kudu/127.2.193.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab.
I20260924 10:39:25.800073 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.787128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38538)
0924 10:39:25.787316 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.787334 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:25.787415 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:25.788131 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:25.788140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:25.788292 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:25.788383 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.788388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.789968 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.789970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.790517 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.790525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.791409 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.791412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.791517 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.792082 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:25.792099 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:25.795748 (+ 3649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:25.797996 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.798002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.798007 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.798212 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.798511 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.798547 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.798548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.798665 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.799186 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:25.799191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:25.799299 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 10:39:25.799736 (+ 437us) client_negotiation.cc:240] Negotiation successful
0924 10:39:25.799865 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 10:39:25.800318 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.787210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.2.193.189:38791)
0924 10:39:25.787633 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:25.787646 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:25.787673 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:39:25.787739 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:25.787747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:25.787970 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:25.788141 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:25.788502 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.789840 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.790639 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.791277 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.791392 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.795895 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:25.795906 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:25.795923 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:25.795951 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:25.797870 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.798324 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.798331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.798332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.798414 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.798770 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.798772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.798774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.799069 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:25.799183 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:25.799562 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 10:39:25.799807 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260924 10:39:25.800631 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:40365
--webserver_interface=127.2.193.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab
--principal=kudu/127.2.193.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:37411,127.2.193.189:38791,127.2.193.188:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:39:25.802732 25237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:25.802796 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.792603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38554)
0924 10:39:25.792718 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.792735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:25.792818 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:25.793386 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:25.793389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:25.793558 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:25.793692 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.793698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.794466 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.794470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.795502 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.795515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.796082 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.796085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.796175 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.796725 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:25.796737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:25.798220 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:25.801027 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.801034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.801039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.801346 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.801652 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:25.801655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:25.801657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:25.801751 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:25.802234 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:25.802239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:25.802360 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 10:39:25.802462 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 10:39:25.802629 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 10:39:25.803452 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.803209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:51956)
0924 10:39:25.803329 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.803395 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:39:25.803586 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.792708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.2.193.189:38791)
0924 10:39:25.793103 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:25.793115 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:25.793129 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:39:25.793213 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:25.793219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:25.793268 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:25.793364 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:25.793807 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.794348 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.795616 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.795960 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.796060 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.798355 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:25.798359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:25.798361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:25.798385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:25.800927 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.801462 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.801466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.801467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.801556 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:25.801854 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:25.801857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:25.801858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:25.802088 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:25.803130 (+ 1042us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:25.803207 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 10:39:25.803331 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":140,"threads_started":1}
W20260924 10:39:25.803671 25186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
I20260924 10:39:25.804409 25162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:25.804677 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.804387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:51972)
0924 10:39:25.804536 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.804589 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 10:39:25.804888 25237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:40365: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
I20260924 10:39:25.805260 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.804888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:51986)
0924 10:39:25.805039 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.805100 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 10:39:25.806005 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.805693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:51998)
0924 10:39:25.805865 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.805925 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 10:39:25.806144 25162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:40365: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
I20260924 10:39:25.837323 25162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } attempt: 1
I20260924 10:39:25.838253 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.837900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52004)
0924 10:39:25.838079 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.838153 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 10:39:25.839010 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.838772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52012)
0924 10:39:25.838893 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.838945 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:39:25.839170 25162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:40365: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
I20260924 10:39:25.854172 25237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } attempt: 1
I20260924 10:39:25.855100 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.854750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52014)
0924 10:39:25.854930 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.855008 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 10:39:25.855880 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.855654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52022)
0924 10:39:25.855763 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.855817 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:39:25.856040 25237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:40365: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
I20260924 10:39:25.914362 25162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } attempt: 2
I20260924 10:39:25.915236 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.914882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52024)
0924 10:39:25.915063 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.915136 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:39:25.916036 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.915782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52038)
0924 10:39:25.915893 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.915942 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:39:25.916172 25162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:40365: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
W20260924 10:39:25.918267 25244 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:39:25.918540 25244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:25.918609 25244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:25.918728 25244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:25.922797 25244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:25.922847 25244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:25.922868 25244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:25.922911 25244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:25.922945 25244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:25.924206 25237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } attempt: 2
I20260924 10:39:25.924939 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.924650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52054)
0924 10:39:25.924803 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.924864 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:39:25.925635 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.925438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52066)
0924 10:39:25.925535 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.925581 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:39:25.925827 25237 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:40365: Network error: Client connection negotiation failed: client connection to 127.2.193.188:40365: connect: Connection refused (error 111)
I20260924 10:39:25.928773 25244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:37411,127.2.193.189:38791,127.2.193.188:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab
--principal=kudu/127.2.193.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.193.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:25.929884 25244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:25.930889 25244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:25.939127 25251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:25.939260 25244 server_base.cc:1034] running on GCE node
W20260924 10:39:25.939147 25250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:25.939147 25253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:25.940882 25244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:25.942087 25244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:25.943281 25244 hybrid_clock.cc:648] HybridClock initialized: now 1790246365943266 us; error 26 us; skew 500 ppm
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:25.946125 25244 init.cc:377] Logged in from keytab as kudu/127.2.193.188@KRBTEST.COM (short username kudu)
I20260924 10:39:25.947078 25244 webserver.cc:460] Webserver started at http://127.2.193.188:39595/ using document root <none> and password file <none>
I20260924 10:39:25.947682 25244 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:25.947781 25244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:25.948020 25244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:25.950270 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data/instance:
uuid: "e25dc5981e5b4977821884bf79e61da2"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "af96ee46094c6123db17aa89f92e04c2"
server_key_iv: "decdac60bf4a13ec17386c707eb49ea3"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.950819 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/wal/instance:
uuid: "e25dc5981e5b4977821884bf79e61da2"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "af96ee46094c6123db17aa89f92e04c2"
server_key_iv: "decdac60bf4a13ec17386c707eb49ea3"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.954563 25244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:25.957026 25260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:25.958319 25244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:25.958452 25244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/wal
uuid: "e25dc5981e5b4977821884bf79e61da2"
format_stamp: "Formatted at 2026-09-24 10:39:25 on dist-test-slave-8wc7"
server_key: "af96ee46094c6123db17aa89f92e04c2"
server_key_iv: "decdac60bf4a13ec17386c707eb49ea3"
server_key_version: "encryptionkey@0"
I20260924 10:39:25.958573 25244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:25.975272 25244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:25.978276 25244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:25.978497 25244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:25.985437 25244 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.188:40365
I20260924 10:39:25.985724 25312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.188:40365 every 8 connection(s)
I20260924 10:39:25.986460 25244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
I20260924 10:39:25.989019 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 25244
I20260924 10:39:25.989147 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246085895225-2822-0/minicluster-data/master-2/wal/instance
I20260924 10:39:25.989449 25313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:25.989503 2822 external_mini_cluster.cc:1428] Setting key 85bcc46c23664b09f13d80a3d3042ee8
I20260924 10:39:25.994557 25313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:39:25 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
Sep 24 10:39:26 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:39:26.005071 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.990912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.2.193.188:40365)
0924 10:39:25.991328 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:25.991343 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:25.991377 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:39:25.992553 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:25.992577 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:25.993109 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:25.993510 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:25.994316 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.995397 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.996282 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.996962 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.997109 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:25.999869 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:25.999874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:25.999883 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:25.999909 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.001842 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.002345 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.002351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.002357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.002495 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.002782 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.002785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.002786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.003249 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.003418 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.003778 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.004102 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260924 10:39:26.006055 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.996305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.2.193.190:37411)
0924 10:39:25.996453 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:25.996457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:25.996746 (+ 289us) server_negotiation.cc:407] Connection header received
0924 10:39:25.996923 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:25.996925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:25.996969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:25.997056 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:25.998332 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.998924 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.999874 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.000092 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.000216 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.002760 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.002765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.002770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.002808 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.004357 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.004826 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.004829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.004830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.004902 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.005215 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.005219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.005221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.005439 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.005516 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.005848 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.005946 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:39:26.006143 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:25.996223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:37411 (local address 127.0.0.1:35238)
0924 10:39:25.996627 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:25.996666 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:25.996800 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:25.997501 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:25.997505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:25.997995 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:25.998182 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.998193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:25.999053 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:25.999056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:25.999748 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:25.999758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.000229 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.000232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.000372 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.000816 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:26.000856 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:26.002609 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:26.004458 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.004465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.004470 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.004731 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.004984 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.004987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.004988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.005080 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.005537 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:26.005542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:26.005653 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 10:39:26.005794 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 10:39:26.005957 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":136,"threads_started":1}
I20260924 10:39:26.007279 25313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:39:26 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
Sep 24 10:39:26 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:39:26.015324 25162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } attempt: 3
I20260924 10:39:26.016981 25237 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } attempt: 3
I20260924 10:39:26.017313 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.007884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:38791 (local address 127.0.0.1:38568)
0924 10:39:26.007989 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:26.008003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:26.008066 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:26.008361 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:26.008366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:26.008552 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:26.008666 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.008672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.009724 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.009727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.010559 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.010567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.011066 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.011069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.011149 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.011617 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:26.011637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:26.013219 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:26.015010 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.015017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.015019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.015229 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.015985 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.015988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.015990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.016082 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.016645 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:26.016648 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:26.016743 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 10:39:26.016885 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:39:26.017081 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 10:39:26.017342 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.008014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.2.193.189:38791)
0924 10:39:26.008150 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.008154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.008167 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:26.008213 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.008215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.008251 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.008314 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.008791 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.009619 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.010691 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.010965 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.011050 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.013343 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.013353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.013356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.013379 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.014883 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.015593 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.015596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.015597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.015671 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.016212 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.016216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.016217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.016485 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.016556 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.016953 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.017092 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:39:26.018286 25313 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:26.019181 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.007641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.2.193.190:37411)
0924 10:39:26.007781 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.007784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.007796 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:39:26.007913 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.007918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.007964 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.008047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.008791 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.009594 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.011146 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.011348 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.011459 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.014077 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.014087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.014090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.014155 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.015655 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.016231 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.016234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.016236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.016312 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.016612 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.016615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.016617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.016827 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.017043 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.018885 (+ 1842us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.019016 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 10:39:26 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
Sep 24 10:39:26 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
I20260924 10:39:26.025071 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.015780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52080)
0924 10:39:26.015879 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:26.015893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:26.015960 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:26.016189 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:26.016192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:26.016352 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:26.016504 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.016511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.017955 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.017958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.018767 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.018776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.019414 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.019418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.019497 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.019934 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:26.019947 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:26.021465 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:26.023369 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.023375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.023377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.023616 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.023954 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.023957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.023959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.024057 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.024529 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:26.024532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:26.024637 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 10:39:26.024752 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:39:26.024898 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 10:39:26.025071 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.015866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.2.193.188:40365)
0924 10:39:26.015990 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.015996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.016010 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:39:26.016044 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.016046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.016081 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.016178 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.017095 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.017825 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.018897 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.019273 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.019713 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.021594 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.021597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.021601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.021623 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.023243 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.023759 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.023763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.023768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.023849 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.024177 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.024180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.024182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.024402 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.024535 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.024813 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.024921 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33,"spinlock_wait_cycles":10496}
Sep 24 10:39:26 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
Sep 24 10:39:26 dist-test-slave-8wc7 krb5kdc[25081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
I20260924 10:39:26.028353 25162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d: Bootstrap starting.
I20260924 10:39:26.030284 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.017355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52090)
0924 10:39:26.017450 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:26.017465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:26.017539 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:26.018213 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:26.018216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:26.018387 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:26.018480 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.018486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.020264 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.020268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.021217 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.021228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.021921 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.021925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.022003 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.022423 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:26.022437 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:26.024071 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:26.028425 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.028428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.028430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.028695 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.029052 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.029058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.029060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.029165 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.029733 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:26.029736 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:26.029842 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 10:39:26.029935 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:39:26.030082 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 10:39:26.030654 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.018782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:40365 (local address 127.0.0.1:52098)
0924 10:39:26.018888 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:26.018900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:26.018954 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:26.020317 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:26.020323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:26.020489 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:26.020581 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.020588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.022890 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.022893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.023675 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:26.023688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.024255 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.024259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:26.024338 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.025385 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:26.025398 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:26.026906 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:26.029311 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.029313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.029316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.029525 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.029790 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:26.029794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:26.029797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:26.029873 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:26.030251 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:26.030254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:26.030323 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 10:39:26.030391 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:39:26.030509 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 10:39:26.030992 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.017588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.2.193.188:40365)
0924 10:39:26.017979 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.017988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.017999 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:39:26.018028 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.018031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.018069 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.018582 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.018660 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.020122 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.021355 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.021782 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.022243 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.024218 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.024224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.024228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.024248 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.028311 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.028833 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.028837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.028840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.028930 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.029333 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.029339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.029343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.029594 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.030582 (+ 988us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.030668 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.030799 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":329,"thread_start_us":139,"threads_started":1}
I20260924 10:39:26.031241 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.018927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.2.193.188:40365)
0924 10:39:26.020100 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.020109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.020122 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:26.020155 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.020158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.020195 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.021149 (+ 954us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.021313 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.022791 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.023799 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.024151 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.027121 (+ 2970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.027241 (+ 120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.027244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.027247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.027263 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.029206 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.029625 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.029628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.029629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.029700 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.029963 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.029967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.029970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.030154 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.030903 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.030969 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.031085 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1108,"thread_start_us":85,"threads_started":1}
I20260924 10:39:26.031570 25162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:26.032384 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.020689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.2.193.189:38791)
0924 10:39:26.020864 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.020870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.020884 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:39:26.020913 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.020915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.020958 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.021043 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.021822 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.022641 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.023765 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.024125 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.024237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.028309 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.028313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.028316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.028346 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.030098 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.030584 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.030588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.030589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.030672 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.030973 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.030976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.030978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.031216 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.031427 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.032095 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.032228 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 10:39:26.032485 25162 log.cc:824] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:26.034070 25313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: Bootstrap starting.
I20260924 10:39:26.034112 25237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: Bootstrap starting.
I20260924 10:39:26.035413 25162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d: No bootstrap required, opened a new log
I20260924 10:39:26.036279 25313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:26.036513 25237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:26.036953 25313 log.cc:824] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:26.037314 25237 log.cc:824] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:26.039428 25237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: No bootstrap required, opened a new log
I20260924 10:39:26.039434 25313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: No bootstrap required, opened a new log
I20260924 10:39:26.039671 25162 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.039906 25162 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9e39e6e3424f2a9abf793cce305c7d, State: Initialized, Role: FOLLOWER
I20260924 10:39:26.040232 25162 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.040827 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:26.041652 25162 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:26.042207 25136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dc9e39e6e3424f2a9abf793cce305c7d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35248
I20260924 10:39:26.042443 25136 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:39:26.042557 25136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [term 0 FOLLOWER]: Advancing to term 1
W20260924 10:39:26.042622 25115 tablet.cc:2369] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:39:26.044014 25136 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.044236 25237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.044499 25237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af2a5b3d414484f876fdead7fcc5dd7, State: Initialized, Role: FOLLOWER
I20260924 10:39:26.044788 25237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.045076 25136 leader_election.cc:288] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [CANDIDATE]: Term 1 election: Requested vote from peers 9af2a5b3d414484f876fdead7fcc5dd7 (127.2.193.189:38791), e25dc5981e5b4977821884bf79e61da2 (127.2.193.188:40365)
I20260924 10:39:26.044929 25313 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.045249 25313 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25dc5981e5b4977821884bf79e61da2, State: Initialized, Role: FOLLOWER
I20260924 10:39:26.045579 25313 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.045738 25334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.045910 25334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:26.047030 25237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:26.047027 25313 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:26.047111 25287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e25dc5981e5b4977821884bf79e61da2"
I20260924 10:39:26.047365 25287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:26.048022 25211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9af2a5b3d414484f876fdead7fcc5dd7"
I20260924 10:39:26.048261 25211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:26.048964 25287 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9e39e6e3424f2a9abf793cce305c7d in term 1.
I20260924 10:39:26.049782 25211 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc9e39e6e3424f2a9abf793cce305c7d in term 1.
I20260924 10:39:26.049784 25323 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.049947 25323 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:26.050277 25113 leader_election.cc:302] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9af2a5b3d414484f876fdead7fcc5dd7, dc9e39e6e3424f2a9abf793cce305c7d; no voters:
I20260924 10:39:26.050803 25323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:26.051009 25323 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [term 1 LEADER]: Becoming Leader. State: Replica: dc9e39e6e3424f2a9abf793cce305c7d, State: Running, Role: LEADER
I20260924 10:39:26.052285 25323 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } }
I20260924 10:39:26.051358 25244 master_runner.cc:186] Error getting master registration for 127.2.193.190:37411: OK, instance_id { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" instance_seqno: 1790246365546298 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:39:26.054832 25244 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 10:39:26.055365 25361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 10:39:26.055384 25363 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9e39e6e3424f2a9abf793cce305c7d. Latest consensus state: current_term: 1 leader_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.055514 25363 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: This master's current role is: LEADER
W20260924 10:39:26.055554 25361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 10:39:26.058653 25362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:26.058820 25362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:26.059172 25335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.059389 25335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 10:39:26.063091 25360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:26.064882 25360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:26.069538 25369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:26.070797 25369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:26.075845 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.051231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.2.193.190:37411)
0924 10:39:26.051493 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.051500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.060084 (+ 8584us) server_negotiation.cc:407] Connection header received
0924 10:39:26.063543 (+ 3459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.063547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.063604 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.063657 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.064888 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.065755 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.067020 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.067369 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.067474 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.069787 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.069791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.069793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.069820 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.071567 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.074424 (+ 2857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.074427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.074429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.074539 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.074866 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.074869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.074871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.075083 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.075195 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.075537 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.075686 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260924 10:39:26.083343 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.052102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.2.193.189:38791)
0924 10:39:26.052209 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.052214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.069947 (+ 17733us) server_negotiation.cc:407] Connection header received
0924 10:39:26.071447 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.071451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.071508 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.071561 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.072589 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.073350 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.077218 (+ 3868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.077564 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.077663 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.079511 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.079515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.079518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.079548 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.081455 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.081976 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.081980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.081982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.082064 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.082355 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.082358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.082360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.082574 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.082692 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.083049 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.083178 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:39:26.086448 25287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [term 1 FOLLOWER]: Refusing update from remote peer dc9e39e6e3424f2a9abf793cce305c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:26.087313 25211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [term 1 FOLLOWER]: Refusing update from remote peer dc9e39e6e3424f2a9abf793cce305c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:26.088124 25323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:26.089120 25323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:26.093288 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:26.051672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.2.193.188:40365)
0924 10:39:26.051781 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.051785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.070171 (+ 18386us) server_negotiation.cc:407] Connection header received
0924 10:39:26.070357 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.070363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.070415 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.070469 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.071957 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.072808 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.074025 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:26.074402 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:26.074476 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:26.089310 (+ 14834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:26.089314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:26.089317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:26.089350 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:26.091241 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.091792 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.091796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.091798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.091884 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:26.092199 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:26.092203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:26.092205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:26.092448 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:26.092569 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:26.092920 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 10:39:26.093095 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:39:26.093645 25334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9e39e6e3424f2a9abf793cce305c7d. Latest consensus state: current_term: 1 leader_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.093787 25334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:26.098220 25335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9e39e6e3424f2a9abf793cce305c7d. Latest consensus state: current_term: 1 leader_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.098336 25335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:26.101702 25363 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.101843 25363 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:26.101960 25372 mvcc.cc:205] Tried to move back new op lower bound from 7332849115443429376 to 7332849115354140672. Current Snapshot: MvccSnapshot[applied={T|T < 7332849115443429376}]
I20260924 10:39:26.102286 25334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.102401 25334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:26.102919 25373 mvcc.cc:205] Tried to move back new op lower bound from 7332849115443429376 to 7332849115354140672. Current Snapshot: MvccSnapshot[applied={T|T < 7332849115443429376}]
I20260924 10:39:26.109594 25323 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.109726 25323 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:26.110906 25335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9e39e6e3424f2a9abf793cce305c7d" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 37411 } } peers { permanent_uuid: "9af2a5b3d414484f876fdead7fcc5dd7" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 38791 } } peers { permanent_uuid: "e25dc5981e5b4977821884bf79e61da2" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 40365 } } }
I20260924 10:39:26.111037 25335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:26.111822 25375 mvcc.cc:205] Tried to move back new op lower bound from 7332849115443429376 to 7332849115354140672. Current Snapshot: MvccSnapshot[applied={T|T < 7332849115443429376}]
I20260924 10:39:26.113065 25369 catalog_manager.cc:1347] Generated new cluster ID: 17ca1ffbca414c07a2e1cc8e0f5d863c
I20260924 10:39:26.113134 25369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:26.141156 25369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:26.142717 25369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:26.167624 25369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc9e39e6e3424f2a9abf793cce305c7d: Generated new TSK 0
I20260924 10:39:26.168743 25369 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:39:26.199108 25166 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.196045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.2.193.190:37411)
0924 10:39:26.196192 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.196197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.197966 (+ 1769us) server_negotiation.cc:407] Connection header received
0924 10:39:26.198177 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.198180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.198232 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.198288 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.198956 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35270: BlockingRecv error: recv got EOF from 127.0.0.1:35270 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 10:39:26.200686 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.196579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.2.193.189:38791)
0924 10:39:26.196706 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.196711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.199804 (+ 3093us) server_negotiation.cc:407] Connection header received
0924 10:39:26.199973 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.199976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.200024 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.200075 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.200555 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38594: BlockingRecv error: recv got EOF from 127.0.0.1:38594 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 10:39:26.202219 25318 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.197435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.2.193.188:40365)
0924 10:39:26.197548 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.197553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.201328 (+ 3775us) server_negotiation.cc:407] Connection header received
0924 10:39:26.201495 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.201499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.201548 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.201610 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.202098 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52110: BlockingRecv error: recv got EOF from 127.0.0.1:52110 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 10:39:26.206988 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.205427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.2.193.189:38791)
0924 10:39:26.205537 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.205540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.205987 (+ 447us) server_negotiation.cc:407] Connection header received
0924 10:39:26.206209 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.206213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.206291 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.206343 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.206875 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38604: BlockingRecv error: recv got EOF from 127.0.0.1:38604 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 10:39:26.215946 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.214147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38616 (local address 127.2.193.189:38791)
0924 10:39:26.214269 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.214273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.214866 (+ 593us) server_negotiation.cc:407] Connection header received
0924 10:39:26.215077 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.215080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.215135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.215188 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.215839 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38616: BlockingRecv error: recv got EOF from 127.0.0.1:38616 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 10:39:26.222436 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.220743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.2.193.189:38791)
0924 10:39:26.220856 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.220860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.221433 (+ 573us) server_negotiation.cc:407] Connection header received
0924 10:39:26.221652 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.221655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.221708 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.221763 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.222313 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38624: BlockingRecv error: recv got EOF from 127.0.0.1:38624 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:39:26.232028 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.229386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.2.193.189:38791)
0924 10:39:26.229539 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.229544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.230967 (+ 1423us) server_negotiation.cc:407] Connection header received
0924 10:39:26.231193 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.231196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.231255 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.231308 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.231903 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38634: BlockingRecv error: recv got EOF from 127.0.0.1:38634 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 10:39:26.257325 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.254399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.2.193.189:38791)
0924 10:39:26.254538 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.254543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.255217 (+ 674us) server_negotiation.cc:407] Connection header received
0924 10:39:26.255459 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.255463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.255537 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.255619 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.257140 (+ 1521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38636: BlockingRecv error: recv got EOF from 127.0.0.1:38636 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 10:39:26.277501 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.276035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.2.193.189:38791)
0924 10:39:26.276164 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.276171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.276494 (+ 323us) server_negotiation.cc:407] Connection header received
0924 10:39:26.276683 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.276687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.276748 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.276827 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.277374 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38650: BlockingRecv error: recv got EOF from 127.0.0.1:38650 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:39:26.312809 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.311496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.2.193.189:38791)
0924 10:39:26.311680 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.311685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.311828 (+ 143us) server_negotiation.cc:407] Connection header received
0924 10:39:26.312016 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.312019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.312075 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.312169 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.312690 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38660: BlockingRecv error: recv got EOF from 127.0.0.1:38660 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 10:39:26.380201 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.379040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.2.193.189:38791)
0924 10:39:26.379199 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.379202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.379353 (+ 151us) server_negotiation.cc:407] Connection header received
0924 10:39:26.379482 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.379484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.379533 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.379620 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.380094 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38672: BlockingRecv error: recv got EOF from 127.0.0.1:38672 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 10:39:26.511634 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.510433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.2.193.189:38791)
0924 10:39:26.510590 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.510594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.510795 (+ 201us) server_negotiation.cc:407] Connection header received
0924 10:39:26.510916 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.510919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.510974 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.511056 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.511511 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38686: BlockingRecv error: recv got EOF from 127.0.0.1:38686 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 10:39:26.771236 25243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:26.770140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.2.193.189:38791)
0924 10:39:26.770284 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:26.770287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:26.770366 (+ 79us) server_negotiation.cc:407] Connection header received
0924 10:39:26.770499 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:26.770502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:26.770548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:26.770632 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:26.771123 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38690: BlockingRecv error: recv got EOF from 127.0.0.1:38690 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:39:27.056883 25361 catalog_manager.cc:1259] Loaded cluster ID: 17ca1ffbca414c07a2e1cc8e0f5d863c
I20260924 10:39:27.056977 25361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: loading cluster ID for follower catalog manager: success
I20260924 10:39:27.058772 25361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: acquiring CA information for follower catalog manager: success
I20260924 10:39:27.060005 25361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9af2a5b3d414484f876fdead7fcc5dd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:39:27.066231 25360 catalog_manager.cc:1259] Loaded cluster ID: 17ca1ffbca414c07a2e1cc8e0f5d863c
I20260924 10:39:27.066301 25360 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: loading cluster ID for follower catalog manager: success
I20260924 10:39:27.067956 25360 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: acquiring CA information for follower catalog manager: success
I20260924 10:39:27.069116 25360 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e25dc5981e5b4977821884bf79e61da2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:39:27.289846 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:27.285581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.2.193.189:38791)
0924 10:39:27.285972 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:27.285978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:27.286020 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:39:27.286216 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:27.286219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:27.286350 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:27.286482 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:27.286757 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.287299 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.288204 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.288445 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.288560 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:27.288891 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:27.289421 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:27.289557 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:27.289621 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 10:39:27.289682 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":66,"threads_started":1}
I20260924 10:39:27.296929 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:27.292986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.2.193.188:40365)
0924 10:39:27.293253 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:27.293258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:27.293408 (+ 150us) server_negotiation.cc:407] Connection header received
0924 10:39:27.293601 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:27.293604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:27.293724 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:27.293849 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:27.294078 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.294588 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.295444 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.295684 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.295796 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:27.296099 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:27.296525 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:27.296674 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:27.296710 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 10:39:27.296767 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":59,"threads_started":1}
I20260924 10:39:27.304036 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:27.300216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.2.193.188:40365)
0924 10:39:27.300334 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:27.300339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:27.300852 (+ 513us) server_negotiation.cc:407] Connection header received
0924 10:39:27.300897 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:27.300900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:27.300965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:27.301037 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:27.301337 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.301833 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.302836 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.303130 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.303230 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:27.303551 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:27.303639 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:27.303702 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:27.303837 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 10:39:27.303894 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:39:27.304147 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:27.299869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.2.193.189:38791)
0924 10:39:27.300010 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:27.300017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:27.300288 (+ 271us) server_negotiation.cc:407] Connection header received
0924 10:39:27.300457 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:27.300460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:27.300516 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:27.300607 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:27.300834 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.301594 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.302861 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.303158 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.303255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:27.303634 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:27.303714 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:27.303783 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:27.303941 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:39:27.304004 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:39:27.304631 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:27.299817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.2.193.190:37411)
0924 10:39:27.300075 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:27.300079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:27.300299 (+ 220us) server_negotiation.cc:407] Connection header received
0924 10:39:27.300406 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:27.300411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:27.300600 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:27.300719 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:27.301071 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.301849 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.303021 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:27.303356 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:27.303473 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:27.303791 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:27.304139 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:27.304231 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:27.304414 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 10:39:27.304485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":63,"threads_started":1}
I20260924 10:39:27.317479 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 25094
I20260924 10:39:27.323755 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 25169
I20260924 10:39:27.329775 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 25244
2026-09-24T10:39:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3133 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:27 dist-test-slave-8wc7 krb5kdc[25462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:27 dist-test-slave-8wc7 krb5kdc[25462](info): set up 2 sockets
Sep 24 10:39:27 dist-test-slave-8wc7 krb5kdc[25462](info): commencing operation
krb5kdc: starting...
W20260924 10:39:28.369649 2822 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/test-user.keytab.
Sep 24 10:39:28 dist-test-slave-8wc7 krb5kdc[25462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.193.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
Entry for principal kudu/127.2.193.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab.
I20260924 10:39:28.500612 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:42737
--webserver_interface=127.2.193.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:42737,127.2.193.189:46073,127.2.193.188:46263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:28.615008 25475 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:39:28.615332 25475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:28.615360 25475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:28.615398 25475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:28.619558 25475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:28.619629 25475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:28.619648 25475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:28.619668 25475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:28.619683 25475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:28.625231 25475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:42737,127.2.193.189:46073,127.2.193.188:46263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.190.keytab
--principal=kudu/127.2.193.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.190:42737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.193.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:28.626356 25475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:28.627377 25475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:28.636307 25481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:28.636330 25480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:28.636384 25475 server_base.cc:1034] running on GCE node
W20260924 10:39:28.636323 25483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:28.637212 25475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:28.638343 25475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:28.639518 25475 hybrid_clock.cc:648] HybridClock initialized: now 1790246368639502 us; error 29 us; skew 500 ppm
Sep 24 10:39:28 dist-test-slave-8wc7 krb5kdc[25462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:28.642428 25475 init.cc:377] Logged in from keytab as kudu/127.2.193.190@KRBTEST.COM (short username kudu)
I20260924 10:39:28.643311 25475 webserver.cc:460] Webserver started at http://127.2.193.190:39119/ using document root <none> and password file <none>
I20260924 10:39:28.643842 25475 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:28.643901 25475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:28.644083 25475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:28.646239 25475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data/instance:
uuid: "5ed927302f7b4874b95cec3aaf8a012a"
format_stamp: "Formatted at 2026-09-24 10:39:28 on dist-test-slave-8wc7"
server_key: "555934b0b014887f50e03d0175b317d4"
server_key_iv: "9cafaa5c71f141a7427de71aa09d3c72"
server_key_version: "encryptionkey@0"
I20260924 10:39:28.646838 25475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/wal/instance:
uuid: "5ed927302f7b4874b95cec3aaf8a012a"
format_stamp: "Formatted at 2026-09-24 10:39:28 on dist-test-slave-8wc7"
server_key: "555934b0b014887f50e03d0175b317d4"
server_key_iv: "9cafaa5c71f141a7427de71aa09d3c72"
server_key_version: "encryptionkey@0"
I20260924 10:39:28.650701 25475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:28.653111 25490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:28.654423 25475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:28.654567 25475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/wal
uuid: "5ed927302f7b4874b95cec3aaf8a012a"
format_stamp: "Formatted at 2026-09-24 10:39:28 on dist-test-slave-8wc7"
server_key: "555934b0b014887f50e03d0175b317d4"
server_key_iv: "9cafaa5c71f141a7427de71aa09d3c72"
server_key_version: "encryptionkey@0"
I20260924 10:39:28.654672 25475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:28.671597 25475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:28.674695 25475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:28.674901 25475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:28.682224 25475 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.190:42737
I20260924 10:39:28.682507 25542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.190:42737 every 8 connection(s)
I20260924 10:39:28.683260 25475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:28.686234 25543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:28.687481 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 25475
I20260924 10:39:28.687633 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-0/wal/instance
I20260924 10:39:28.687948 2822 external_mini_cluster.cc:1428] Setting key 7f731e9a9a3ea2557aca172b5f993dfe
I20260924 10:39:28.690773 25543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.193.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.189@KRBTEST.COM" created.
Sep 24 10:39:28 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:39:28.707630 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.693838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.2.193.190:42737)
0924 10:39:28.694299 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:28.694320 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:28.694592 (+ 272us) server_negotiation.cc:407] Connection header received
0924 10:39:28.695689 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:28.695694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:28.695891 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:28.696089 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:28.696937 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.698007 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.699020 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.699507 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.699694 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.702409 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:28.702415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:28.702425 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:28.702446 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:28.704261 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.704766 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.704779 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.704780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.704886 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.705228 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.705233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.705236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.705647 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:28.705795 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:28.706187 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 10:39:28.706459 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":141,"threads_started":1}
I20260924 10:39:28.707630 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.693840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42737 (local address 127.0.0.1:46674)
0924 10:39:28.694443 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.694482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:28.695355 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:28.696086 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:28.696091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:28.696527 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:28.696787 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.696800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.698150 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.698159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.698898 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.698907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.699685 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.699689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.699814 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.700427 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:28.700445 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:28.702236 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:28.704388 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.704399 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.704412 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.704651 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.704991 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.704993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.704996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.705112 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.705772 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:28.705779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:28.705927 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 10:39:28.706112 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 10:39:28.706513 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":113,"thread_start_us":195,"threads_started":1}
I20260924 10:39:28.710821 25543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:28.711655 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.711346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39148)
0924 10:39:28.711475 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.711547 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 10:39:28.711911 25494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
I20260924 10:39:28.712872 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.712612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39154)
0924 10:39:28.712748 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.712814 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 10:39:28.713037 25543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:46073: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab.
Entry for principal kudu/127.2.193.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab.
I20260924 10:39:28.716734 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:46073
--webserver_interface=127.2.193.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab
--principal=kudu/127.2.193.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:42737,127.2.193.189:46073,127.2.193.188:46263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:39:28.762306 25543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } attempt: 1
I20260924 10:39:28.763389 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.763054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39166)
0924 10:39:28.763236 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.763306 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 10:39:28.764220 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.763995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39168)
0924 10:39:28.764108 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.764160 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:39:28.764353 25543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:46073: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
I20260924 10:39:28.832508 25543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } attempt: 2
I20260924 10:39:28.833328 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.833010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39176)
0924 10:39:28.833161 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.833251 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 10:39:28.833827 25550 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260924 10:39:28.834052 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.833833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39184)
0924 10:39:28.833945 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.833997 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:39:28.834165 25550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:28.834194 25543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.189:46073: Network error: Client connection negotiation failed: client connection to 127.2.193.189:46073: connect: Connection refused (error 111)
W20260924 10:39:28.834250 25550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:28.834324 25550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:28.838574 25550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:28.838639 25550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:28.838660 25550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:28.838677 25550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:28.838732 25550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:28.844324 25550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:42737,127.2.193.189:46073,127.2.193.188:46263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.189.keytab
--principal=kudu/127.2.193.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.189:46073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.193.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:28.845588 25550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:28.846658 25550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:28.855540 25558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:28.855540 25555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:28.855537 25556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:28.855974 25550 server_base.cc:1034] running on GCE node
I20260924 10:39:28.856568 25550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:28.857818 25550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:28.859025 25550 hybrid_clock.cc:648] HybridClock initialized: now 1790246368858997 us; error 42 us; skew 500 ppm
Sep 24 10:39:28 dist-test-slave-8wc7 krb5kdc[25462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:28.862082 25550 init.cc:377] Logged in from keytab as kudu/127.2.193.189@KRBTEST.COM (short username kudu)
I20260924 10:39:28.863022 25550 webserver.cc:460] Webserver started at http://127.2.193.189:36351/ using document root <none> and password file <none>
I20260924 10:39:28.863624 25550 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:28.863721 25550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:28.863965 25550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:28.866163 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data/instance:
uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a"
format_stamp: "Formatted at 2026-09-24 10:39:28 on dist-test-slave-8wc7"
server_key: "d206197d5a0e9bc7e79cf54033eda76e"
server_key_iv: "71a363bbe5c32d095022158de9ebcf91"
server_key_version: "encryptionkey@0"
I20260924 10:39:28.866773 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/wal/instance:
uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a"
format_stamp: "Formatted at 2026-09-24 10:39:28 on dist-test-slave-8wc7"
server_key: "d206197d5a0e9bc7e79cf54033eda76e"
server_key_iv: "71a363bbe5c32d095022158de9ebcf91"
server_key_version: "encryptionkey@0"
I20260924 10:39:28.870661 25550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:28.873292 25565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:28.874619 25550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:39:28.874776 25550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/wal
uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a"
format_stamp: "Formatted at 2026-09-24 10:39:28 on dist-test-slave-8wc7"
server_key: "d206197d5a0e9bc7e79cf54033eda76e"
server_key_iv: "71a363bbe5c32d095022158de9ebcf91"
server_key_version: "encryptionkey@0"
I20260924 10:39:28.874897 25550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:28.898931 25550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:28.902555 25550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:28.902779 25550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:28.910058 25550 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.189:46073
I20260924 10:39:28.910348 25617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.189:46073 every 8 connection(s)
I20260924 10:39:28.911073 25550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/data/info.pb
I20260924 10:39:28.913630 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 25550
I20260924 10:39:28.913759 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-1/wal/instance
I20260924 10:39:28.914080 2822 external_mini_cluster.cc:1428] Setting key f82c33577024b1edcdb6df6a19c78d44
I20260924 10:39:28.914153 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:28.919390 25618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.193.188@KRBTEST.COM; defaulting to no policy
I20260924 10:39:28.925349 25543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.193.188@KRBTEST.COM" created.
Sep 24 10:39:28 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
Sep 24 10:39:28 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
I20260924 10:39:28.936398 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.925814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39192)
0924 10:39:28.926445 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.926463 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:28.926553 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:28.927034 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:28.927040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:28.927252 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:28.927353 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.927358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.928376 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.928379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.929356 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.929370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.930014 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.930020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.930182 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.930704 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:28.930724 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:28.932192 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:28.934196 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.934201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.934204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.934478 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.934882 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.934885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.934888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.934994 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.935830 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:28.935833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:28.935913 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 10:39:28.935996 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:39:28.936159 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539}
I20260924 10:39:28.936419 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.922006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.2.193.190:42737)
0924 10:39:28.922202 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:28.922208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:28.923061 (+ 853us) server_negotiation.cc:407] Connection header received
0924 10:39:28.924072 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:28.924075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:28.924131 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:28.924216 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:28.925688 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.926418 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.928203 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.928516 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.928633 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.932074 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:28.932080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:28.932083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:28.932121 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:28.934600 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.935112 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.935115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.935117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.935192 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.935456 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.935458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.935459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.935669 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:28.935772 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:28.936208 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 10:39:28.936312 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:39:28.937562 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.925941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.2.193.189:46073)
0924 10:39:28.926320 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:28.926331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:28.926570 (+ 239us) server_negotiation.cc:407] Connection header received
0924 10:39:28.926642 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:28.926647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:28.926851 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:28.927023 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:28.927488 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.928253 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.929496 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.929846 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.929985 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.932335 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:28.932342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:28.932361 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:28.932387 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:28.934074 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.934592 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.934599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.934606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.934774 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.935101 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.935106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.935109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.935708 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:28.935849 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:28.936109 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 10:39:28.936419 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":164,"threads_started":1}
I20260924 10:39:28.937559 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.922353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42737 (local address 127.0.0.1:46688)
0924 10:39:28.922880 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.922932 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:28.923865 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:28.924381 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:28.924388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:28.924779 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:28.925508 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.925533 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.926557 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.926561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.928054 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.928066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.928676 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.928679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.928865 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.930096 (+ 1231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:28.930117 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:28.931868 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:28.934724 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.934732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.934738 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.935006 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.935279 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.935282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.935283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.935360 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.935781 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:28.935791 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:28.935940 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 10:39:28.936120 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 10:39:28.936499 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":86,"thread_start_us":190,"threads_started":1}
I20260924 10:39:28.939011 25618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:28.939862 25543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:28.940557 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.940320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:48964)
0924 10:39:28.940435 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.940487 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:39:28.941268 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.940956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:48972)
0924 10:39:28.941098 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.941158 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 10:39:28.941401 25543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:46263: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.193.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab.
Entry for principal kudu/127.2.193.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab.
I20260924 10:39:28.943809 2822 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBKECxY/build/debug/bin/kudu
/tmp/dist-test-taskBKECxY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:46263
--webserver_interface=127.2.193.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab
--principal=kudu/127.2.193.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.193.148:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.193.190:42737,127.2.193.189:46073,127.2.193.188:46263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:39:28 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
I20260924 10:39:28.948086 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.939542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39208)
0924 10:39:28.939636 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.939652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:28.939749 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:28.940093 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:28.940095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:28.940266 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:28.940404 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.940414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.941418 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.941421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.942065 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:28.942074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.942608 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.942612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:28.942710 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.943251 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:28.943266 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:28.944732 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:28.946575 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.946577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.946579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.946767 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.947050 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:28.947052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:28.947053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:28.947132 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:28.947628 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:28.947630 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:28.947707 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 10:39:28.947785 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:39:28.947924 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 10:39:28.948253 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.939597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.2.193.189:46073)
0924 10:39:28.939901 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:28.939907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:28.939924 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:39:28.939948 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:28.939949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:28.940012 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:28.940084 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:28.940503 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.941229 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.942206 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:28.942467 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:28.942582 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:28.944871 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:28.944878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:28.944884 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:28.944916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:28.946405 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.946862 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.946868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.946873 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.946961 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:28.947217 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:28.947223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:28.947228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:28.947532 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:28.947638 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:28.947873 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:39:28.948003 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260924 10:39:28.949015 25618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:28.949841 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.949594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:48976)
0924 10:39:28.949713 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.949787 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 10:39:28.950380 25566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
I20260924 10:39:28.951216 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.951003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:48986)
0924 10:39:28.951119 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.951169 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:39:28.951380 25618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:46263: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
I20260924 10:39:28.972678 25543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } attempt: 1
I20260924 10:39:28.973645 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.973339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:48990)
0924 10:39:28.973496 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.973560 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:39:28.974392 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:28.974172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49000)
0924 10:39:28.974286 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:28.974340 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:39:28.974529 25543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:46263: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
I20260924 10:39:29.000634 25618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } attempt: 1
I20260924 10:39:29.001610 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.001309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49010)
0924 10:39:29.001465 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.001530 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:39:29.002367 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.002152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49022)
0924 10:39:29.002261 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.002315 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:39:29.002501 25618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:46263: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
I20260924 10:39:29.049718 25543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } attempt: 2
I20260924 10:39:29.050613 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.050300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49032)
0924 10:39:29.050458 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.050530 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:39:29.051365 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.051123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49048)
0924 10:39:29.051228 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.051300 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:39:29.051906 25543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:46263: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
W20260924 10:39:29.060496 25624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:39:29.060833 25624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:29.060899 25624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:29.060981 25624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:29.064975 25624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:29.065068 25624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:29.065109 25624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:29.065145 25624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:29.065193 25624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:29.070636 25618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } attempt: 2
I20260924 10:39:29.071072 25624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.193.148:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.193.190:42737,127.2.193.189:46073,127.2.193.188:46263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/krb5kdc/kudu_127.2.193.188.keytab
--principal=kudu/127.2.193.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.193.188:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.193.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:29.071398 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.071116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49060)
0924 10:39:29.071258 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.071318 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:39:29.072041 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.071836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49062)
0924 10:39:29.071939 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.071983 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:39:29.072183 25618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.193.188:46263: Network error: Client connection negotiation failed: client connection to 127.2.193.188:46263: connect: Connection refused (error 111)
I20260924 10:39:29.072325 25624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:29.073395 25624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:29.081631 25633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:29.081631 25630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:29.081626 25631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:29.082001 25624 server_base.cc:1034] running on GCE node
I20260924 10:39:29.082584 25624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:29.083649 25624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:29.084805 25624 hybrid_clock.cc:648] HybridClock initialized: now 1790246369084769 us; error 49 us; skew 500 ppm
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:29.087529 25624 init.cc:377] Logged in from keytab as kudu/127.2.193.188@KRBTEST.COM (short username kudu)
I20260924 10:39:29.088354 25624 webserver.cc:460] Webserver started at http://127.2.193.188:36303/ using document root <none> and password file <none>
I20260924 10:39:29.088901 25624 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:29.088991 25624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:29.089274 25624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:29.091253 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data/instance:
uuid: "3ef181c7800d4621a0577af40f33cc7e"
format_stamp: "Formatted at 2026-09-24 10:39:29 on dist-test-slave-8wc7"
server_key: "4daf6fdd6348baeca88666d921f77320"
server_key_iv: "2568f56e7e9ecf70de28a49385afee48"
server_key_version: "encryptionkey@0"
I20260924 10:39:29.091805 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/wal/instance:
uuid: "3ef181c7800d4621a0577af40f33cc7e"
format_stamp: "Formatted at 2026-09-24 10:39:29 on dist-test-slave-8wc7"
server_key: "4daf6fdd6348baeca88666d921f77320"
server_key_iv: "2568f56e7e9ecf70de28a49385afee48"
server_key_version: "encryptionkey@0"
I20260924 10:39:29.095512 25624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:39:29.097795 25640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:29.098982 25624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:29.099145 25624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data,/tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/wal
uuid: "3ef181c7800d4621a0577af40f33cc7e"
format_stamp: "Formatted at 2026-09-24 10:39:29 on dist-test-slave-8wc7"
server_key: "4daf6fdd6348baeca88666d921f77320"
server_key_iv: "2568f56e7e9ecf70de28a49385afee48"
server_key_version: "encryptionkey@0"
I20260924 10:39:29.099255 25624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:29.118494 25624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:29.121685 25624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:29.121877 25624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:29.128477 25624 rpc_server.cc:307] RPC server started. Bound to: 127.2.193.188:46263
I20260924 10:39:29.128758 25692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.193.188:46263 every 8 connection(s)
I20260924 10:39:29.129451 25624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/data/info.pb
I20260924 10:39:29.131667 2822 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBKECxY/build/debug/bin/kudu as pid 25624
I20260924 10:39:29.131803 2822 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBKECxY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246085895225-2822-0/minicluster-data/master-2/wal/instance
I20260924 10:39:29.132102 2822 external_mini_cluster.cc:1428] Setting key 678545f7496290c682ac4cf30bdd590a
I20260924 10:39:29.132381 25693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:29.136663 25693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
I20260924 10:39:29.147114 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.133510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.2.193.188:46263)
0924 10:39:29.133926 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.133941 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.133974 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:39:29.134627 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.134642 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.134967 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.135298 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.136106 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.137214 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.138085 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.138745 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.138913 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.141700 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.141706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.141722 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.141759 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.143542 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.144083 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.144088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.144092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.144198 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.144509 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.144513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.144515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.145067 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.145302 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.145712 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.146030 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":130,"threads_started":1}
I20260924 10:39:29.147848 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.138321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.190:42737 (local address 127.0.0.1:46704)
0924 10:39:29.138697 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.138747 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:29.138868 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:29.139439 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:29.139447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:29.139816 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:29.139940 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.139946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.140869 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.140874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.141681 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.141689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.142128 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.142131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.142247 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.142747 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:29.142759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:29.144322 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:29.146281 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.146288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.146294 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.146558 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.146819 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.146822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.146824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.146915 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.147375 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:29.147380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:29.147484 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 10:39:29.147599 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:39:29.147731 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":135,"threads_started":1}
I20260924 10:39:29.147979 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.138390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.2.193.190:42737)
0924 10:39:29.138537 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.138542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.139196 (+ 654us) server_negotiation.cc:407] Connection header received
0924 10:39:29.139243 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.139245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.139288 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.139353 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.140070 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.140738 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.141795 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.142009 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.142115 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.144456 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.144478 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.144484 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.144514 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.146168 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.146663 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.146665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.146667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.146734 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.147012 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.147014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.147015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.147213 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.147322 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.147707 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.147826 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:39:29.149639 25693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:29.151080 25543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } attempt: 3
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.190@KRBTEST.COM
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.190@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
I20260924 10:39:29.159499 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.149574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.2.193.190:42737)
0924 10:39:29.149698 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.149701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.149761 (+ 60us) server_negotiation.cc:407] Connection header received
0924 10:39:29.149888 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.149891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.149934 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.150016 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.150752 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.151470 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.152332 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.152540 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.152652 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.155480 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.155484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.155486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.155512 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.157393 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.157880 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.157884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.157886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.157986 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.158590 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.158595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.158598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.158837 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.159062 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.159200 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.159312 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:39:29.160553 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.150127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.189:46073 (local address 127.0.0.1:39214)
0924 10:39:29.150378 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.150394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:29.150479 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:29.150719 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:29.150722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:29.150878 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:29.150959 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.150964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.151880 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.151886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.152671 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.152679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.153337 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.153342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.153464 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.153943 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:29.153956 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:29.156631 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:29.158762 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.158767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.158772 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.159131 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.159490 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.159494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.159499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.159579 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.160046 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:29.160052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:29.160159 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:39:29.160260 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:39:29.160383 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":20}
I20260924 10:39:29.160868 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.150229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.2.193.189:46073)
0924 10:39:29.150336 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.150339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.150445 (+ 106us) server_negotiation.cc:407] Connection header received
0924 10:39:29.150561 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.150563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.150600 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.150664 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.151066 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.151775 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.152808 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.153167 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.153337 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.156772 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.156775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.156777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.156799 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.158654 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.159255 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.159261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.159262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.159361 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.159684 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.159688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.159690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.159909 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.159975 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.160349 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.160488 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:39:29.160881 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.151456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49070)
0924 10:39:29.151609 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.151620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:29.151695 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:29.151910 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:29.151913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:29.152073 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:29.152153 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.152159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.153438 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.153441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.154252 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.154262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.154823 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.154827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.154914 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.155360 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:29.155381 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:29.156646 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:29.159292 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.159298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.159302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.159568 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.159891 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.159894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.159897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.159992 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.160410 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:29.160413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:29.160505 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 10:39:29.160589 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:39:29.160727 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 10:39:29.161401 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.151490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.2.193.188:46263)
0924 10:39:29.151586 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.151590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.151660 (+ 70us) server_negotiation.cc:407] Connection header received
0924 10:39:29.151761 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.151764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.151797 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.152256 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.152325 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.153010 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.154401 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.154688 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.154781 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.157143 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.157145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.157147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.157208 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.159165 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.159722 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.159727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.159731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.159794 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.160111 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.160114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.160115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.160307 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.161044 (+ 737us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.161119 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.161240 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":33,"spinlock_wait_cycles":6400}
I20260924 10:39:29.161967 25693 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:39:29.163339 25618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } attempt: 3
I20260924 10:39:29.165385 25543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a: Bootstrap starting.
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246369, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.188@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.193.189@KRBTEST.COM
I20260924 10:39:29.168522 25543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:29.169528 25543 log.cc:824] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:39:29 dist-test-slave-8wc7 krb5kdc[25462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.193.189@KRBTEST.COM for kudu/127.2.193.188@KRBTEST.COM
I20260924 10:39:29.170895 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.162448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49082)
0924 10:39:29.162529 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.162543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:29.162609 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:29.162823 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:29.162827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:29.162975 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:29.163059 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.163065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.163963 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.163967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.164657 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.164668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.165204 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.165208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.165277 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.165711 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:29.165724 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:29.167622 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:29.169467 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.169470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.169472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.169700 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.169979 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.169982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.169984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.170057 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.170389 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:29.170392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:29.170469 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 10:39:29.170577 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:39:29.170722 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 10:39:29.170897 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.162507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.2.193.188:46263)
0924 10:39:29.162601 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.162604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.162616 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:39:29.162680 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.162682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.162718 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.162790 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.163194 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.163846 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.164780 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.165051 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.165465 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.167750 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.167753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.167756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.167776 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.169357 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.169806 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.169809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.169811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.169887 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.170148 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.170150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.170151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.170281 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.170388 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.170633 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.170776 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":37}
I20260924 10:39:29.172516 25543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a: No bootstrap required, opened a new log
I20260924 10:39:29.172869 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.160802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.2.193.189:46073)
0924 10:39:29.161241 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.161250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.161263 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:29.161353 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.161356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.161396 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.161504 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.162304 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.163028 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.164020 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.164336 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.164451 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.167864 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.167867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.167870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.167893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.170624 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.171083 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.171089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.171093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.171194 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.171531 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.171535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.171537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.171751 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.171935 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.172553 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.172677 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":159,"threads_started":1}
I20260924 10:39:29.174078 25693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: Bootstrap starting.
I20260924 10:39:29.174829 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.163666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.193.188:46263 (local address 127.0.0.1:49094)
0924 10:39:29.163796 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:29.163808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:29.163869 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:29.164936 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:29.164942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:29.165145 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:29.165270 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.165278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.166539 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.166544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.167127 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:29.167135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.167781 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.167784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:29.167875 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.168416 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:29.168435 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:29.170060 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:29.173339 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.173342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.173344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.173570 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.173872 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:29.173875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:29.173877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:29.173971 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:29.174382 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:29.174385 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:29.174472 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 10:39:29.174546 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:39:29.174670 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 10:39:29.175365 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.164043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.2.193.188:46263)
0924 10:39:29.164690 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.164704 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.164719 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:39:29.164752 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.164756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.164795 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.165388 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.165552 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.166420 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.167240 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.167638 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.168075 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.171463 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.171466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.171468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.171491 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.173218 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.173687 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.173690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.173691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.173770 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.174084 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.174087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.174089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.174272 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.175011 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.175083 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.175207 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":82,"threads_started":1}
I20260924 10:39:29.176147 2822 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:29.176874 25693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:29.177395 25517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5ed927302f7b4874b95cec3aaf8a012a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46712
I20260924 10:39:29.177831 25693 log.cc:824] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:29.178278 25618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: Bootstrap starting.
I20260924 10:39:29.178409 25543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.178638 25543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed927302f7b4874b95cec3aaf8a012a, State: Initialized, Role: FOLLOWER
I20260924 10:39:29.178973 25543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.180172 25693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: No bootstrap required, opened a new log
I20260924 10:39:29.180400 25543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:29.180476 25517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:39:29.180604 25517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:29.181046 25618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:29.181946 25517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.181978 25618 log.cc:824] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:29.182989 25517 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [CANDIDATE]: Term 1 election: Requested vote from peers e138dc73f90142bfa7f1fa0d3cf0d50a (127.2.193.189:46073), 3ef181c7800d4621a0577af40f33cc7e (127.2.193.188:46263)
I20260924 10:39:29.183358 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.183458 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:29.184468 25693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.184772 25693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef181c7800d4621a0577af40f33cc7e, State: Initialized, Role: FOLLOWER
I20260924 10:39:29.185104 25693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.185302 25618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: No bootstrap required, opened a new log
I20260924 10:39:29.186546 25693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:39:29.186671 25714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:29.186735 25714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:29.189749 25618 raft_consensus.cc:348] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.190095 25618 raft_consensus.cc:729] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e138dc73f90142bfa7f1fa0d3cf0d50a, State: Initialized, Role: FOLLOWER
I20260924 10:39:29.190634 25618 consensus_queue.cc:260] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.192180 25667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ed927302f7b4874b95cec3aaf8a012a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ef181c7800d4621a0577af40f33cc7e"
I20260924 10:39:29.192318 25618 sys_catalog.cc:564] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:29.192418 25667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [term 0 FOLLOWER]: Advancing to term 1
W20260924 10:39:29.192584 25731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:29.192646 25731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:29.193877 25667 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed927302f7b4874b95cec3aaf8a012a in term 1.
I20260924 10:39:29.194691 25592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ed927302f7b4874b95cec3aaf8a012a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a"
I20260924 10:39:29.194993 25592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:29.195969 25491 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ef181c7800d4621a0577af40f33cc7e, 5ed927302f7b4874b95cec3aaf8a012a; no voters:
I20260924 10:39:29.196285 25702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:29.196429 25592 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed927302f7b4874b95cec3aaf8a012a in term 1.
I20260924 10:39:29.196671 25702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [term 1 LEADER]: Becoming Leader. State: Replica: 5ed927302f7b4874b95cec3aaf8a012a, State: Running, Role: LEADER
I20260924 10:39:29.197080 25702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } }
I20260924 10:39:29.188246 25475 master_runner.cc:186] Error getting master registration for 127.2.193.189:46073: OK, instance_id { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" instance_seqno: 1790246368908071 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:39:29.198654 25475 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:39:29.199316 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed927302f7b4874b95cec3aaf8a012a. Latest consensus state: current_term: 1 leader_uuid: "5ed927302f7b4874b95cec3aaf8a012a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.199554 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:29.200569 25733 sys_catalog.cc:455] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.200736 25733 sys_catalog.cc:458] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 10:39:29.204820 25744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:29.204911 25744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:29.205983 25550 master_runner.cc:263] No leader master found from master e138dc73f90142bfa7f1fa0d3cf0d50a
I20260924 10:39:29.206058 25550 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:39:29.206432 25624 master_runner.cc:263] No leader master found from master 3ef181c7800d4621a0577af40f33cc7e
I20260924 10:39:29.206511 25624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:39:29.210798 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.199296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.2.193.189:46073)
0924 10:39:29.199420 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.199424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.200507 (+ 1083us) server_negotiation.cc:407] Connection header received
0924 10:39:29.200723 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.200727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.200781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.200835 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.202006 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.202810 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.203971 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.204499 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.204622 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.207027 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.207031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.207034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.207063 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.208905 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.209430 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.209435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.209436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.209525 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.209826 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.209829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.209831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.210032 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.210141 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.210483 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.210628 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:39:29.215308 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.199950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.2.193.190:42737)
0924 10:39:29.200047 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.200052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.206687 (+ 6635us) server_negotiation.cc:407] Connection header received
0924 10:39:29.206784 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.206787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.206843 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.206927 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.207826 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.208601 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.209824 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.210153 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.210222 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.211864 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.211868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.211870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.211899 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.213584 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.214051 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.214055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.214057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.214142 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.214433 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.214437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.214439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.214646 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.214768 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.215024 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.215152 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 10:39:29.217777 25720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.218006 25720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:29.220077 25748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:29.221197 25748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:29.222333 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:29.201394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.2.193.188:46263)
0924 10:39:29.202197 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.202202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.206969 (+ 4767us) server_negotiation.cc:407] Connection header received
0924 10:39:29.207143 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.207146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.207196 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.207251 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.208222 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.209039 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.210141 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:29.210513 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:29.210589 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:29.218484 (+ 7895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:29.218488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:29.218491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:29.218525 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:29.220258 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.220830 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.220834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.220836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.220926 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:29.221280 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:29.221284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:29.221285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:29.221535 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:29.221665 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:29.222027 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 10:39:29.222162 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723}
I20260924 10:39:29.233189 25667 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [term 1 FOLLOWER]: Refusing update from remote peer 5ed927302f7b4874b95cec3aaf8a012a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:29.234145 25745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:29.235934 25592 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [term 1 FOLLOWER]: Refusing update from remote peer 5ed927302f7b4874b95cec3aaf8a012a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:29.236478 25745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:29.238144 25720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed927302f7b4874b95cec3aaf8a012a. Latest consensus state: current_term: 1 leader_uuid: "5ed927302f7b4874b95cec3aaf8a012a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.238262 25720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:29.241640 25745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ed927302f7b4874b95cec3aaf8a012a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.241775 25745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:29.242025 25720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ed927302f7b4874b95cec3aaf8a012a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.242147 25720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:29.243103 25753 mvcc.cc:205] Tried to move back new op lower bound from 7332849128370216960 to 7332849128235499520. Current Snapshot: MvccSnapshot[applied={T|T < 7332849128370216960}]
I20260924 10:39:29.243252 25733 sys_catalog.cc:455] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed927302f7b4874b95cec3aaf8a012a. Latest consensus state: current_term: 1 leader_uuid: "5ed927302f7b4874b95cec3aaf8a012a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.243384 25733 sys_catalog.cc:458] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:29.243525 25752 mvcc.cc:205] Tried to move back new op lower bound from 7332849128370216960 to 7332849128235499520. Current Snapshot: MvccSnapshot[applied={T|T < 7332849128370216960}]
I20260924 10:39:29.252331 25748 catalog_manager.cc:1347] Generated new cluster ID: 4a167db1603e4a79a06493df068b59ba
I20260924 10:39:29.252451 25748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:29.265699 25748 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:29.267225 25748 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:29.274005 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ed927302f7b4874b95cec3aaf8a012a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.274154 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:29.275676 25755 mvcc.cc:205] Tried to move back new op lower bound from 7332849128370216960 to 7332849128235499520. Current Snapshot: MvccSnapshot[applied={T|T < 7332849128370216960}]
I20260924 10:39:29.276281 25733 sys_catalog.cc:455] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ed927302f7b4874b95cec3aaf8a012a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed927302f7b4874b95cec3aaf8a012a" member_type: VOTER last_known_addr { host: "127.2.193.190" port: 42737 } } peers { permanent_uuid: "e138dc73f90142bfa7f1fa0d3cf0d50a" member_type: VOTER last_known_addr { host: "127.2.193.189" port: 46073 } } peers { permanent_uuid: "3ef181c7800d4621a0577af40f33cc7e" member_type: VOTER last_known_addr { host: "127.2.193.188" port: 46263 } } }
I20260924 10:39:29.276417 25733 sys_catalog.cc:458] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:39:29.288040 25748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed927302f7b4874b95cec3aaf8a012a: Generated new TSK 0
I20260924 10:39:29.288978 25748 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:39:29.319849 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.317902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.2.193.189:46073)
0924 10:39:29.318079 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.318084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.318648 (+ 564us) server_negotiation.cc:407] Connection header received
0924 10:39:29.318861 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.318864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.318915 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.318969 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.319700 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39250: BlockingRecv error: recv got EOF from 127.0.0.1:39250 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 10:39:29.322430 25546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.320584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.2.193.190:42737)
0924 10:39:29.320703 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.320708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.321464 (+ 756us) server_negotiation.cc:407] Connection header received
0924 10:39:29.321648 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.321651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.321703 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.321755 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.322297 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46724: BlockingRecv error: recv got EOF from 127.0.0.1:46724 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 10:39:29.323949 25699 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.321136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.2.193.188:46263)
0924 10:39:29.321299 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.321303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.323059 (+ 1756us) server_negotiation.cc:407] Connection header received
0924 10:39:29.323227 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.323231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.323285 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.323335 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.323830 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49126: BlockingRecv error: recv got EOF from 127.0.0.1:49126 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 10:39:29.328554 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.326828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.2.193.189:46073)
0924 10:39:29.326940 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.326945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.327487 (+ 542us) server_negotiation.cc:407] Connection header received
0924 10:39:29.327730 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.327733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.327815 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.327868 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.328441 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39260: BlockingRecv error: recv got EOF from 127.0.0.1:39260 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:39:29.333663 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.332008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.2.193.189:46073)
0924 10:39:29.332112 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.332116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.332643 (+ 527us) server_negotiation.cc:407] Connection header received
0924 10:39:29.332837 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.332841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.332905 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.332957 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.333550 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39268: BlockingRecv error: recv got EOF from 127.0.0.1:39268 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 10:39:29.339465 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.337783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.2.193.189:46073)
0924 10:39:29.337890 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.337894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.338464 (+ 570us) server_negotiation.cc:407] Connection header received
0924 10:39:29.338671 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.338675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.338732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.338788 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.339343 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39270: BlockingRecv error: recv got EOF from 127.0.0.1:39270 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 10:39:29.347922 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.345971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.2.193.189:46073)
0924 10:39:29.346137 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.346142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.346807 (+ 665us) server_negotiation.cc:407] Connection header received
0924 10:39:29.347053 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.347057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.347115 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.347171 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.347808 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39286: BlockingRecv error: recv got EOF from 127.0.0.1:39286 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 10:39:29.361035 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.359034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.2.193.189:46073)
0924 10:39:29.359188 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.359195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.359851 (+ 656us) server_negotiation.cc:407] Connection header received
0924 10:39:29.360090 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.360094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.360153 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.360212 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.360885 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39298: BlockingRecv error: recv got EOF from 127.0.0.1:39298 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 10:39:29.381970 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.379960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.2.193.189:46073)
0924 10:39:29.380105 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.380110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.380805 (+ 695us) server_negotiation.cc:407] Connection header received
0924 10:39:29.381053 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.381058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.381121 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.381205 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.381842 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39314: BlockingRecv error: recv got EOF from 127.0.0.1:39314 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 10:39:29.417765 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.416408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.2.193.189:46073)
0924 10:39:29.416542 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.416546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.416813 (+ 267us) server_negotiation.cc:407] Connection header received
0924 10:39:29.416986 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.416989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.417050 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.417144 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.417645 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39320: BlockingRecv error: recv got EOF from 127.0.0.1:39320 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 10:39:29.485085 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.483741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.2.193.189:46073)
0924 10:39:29.483918 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.483923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.484083 (+ 160us) server_negotiation.cc:407] Connection header received
0924 10:39:29.484268 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.484271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.484332 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.484430 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.484959 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39330: BlockingRecv error: recv got EOF from 127.0.0.1:39330 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 10:39:29.616458 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.615315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.2.193.189:46073)
0924 10:39:29.615491 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.615495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.615541 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:39:29.615691 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.615694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.615743 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.615817 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.616323 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39342: BlockingRecv error: recv got EOF from 127.0.0.1:39342 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 10:39:29.875628 25698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:29.874727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.2.193.189:46073)
0924 10:39:29.874863 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:29.874866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:29.874882 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:39:29.874994 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:29.874997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:29.875043 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:29.875126 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:29.875529 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39352: BlockingRecv error: recv got EOF from 127.0.0.1:39352 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:39:30.194404 25731 catalog_manager.cc:1259] Loaded cluster ID: 4a167db1603e4a79a06493df068b59ba
I20260924 10:39:30.194494 25731 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: loading cluster ID for follower catalog manager: success
I20260924 10:39:30.196708 25731 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: acquiring CA information for follower catalog manager: success
I20260924 10:39:30.197993 25731 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ef181c7800d4621a0577af40f33cc7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:39:30.206329 25744 catalog_manager.cc:1259] Loaded cluster ID: 4a167db1603e4a79a06493df068b59ba
I20260924 10:39:30.206416 25744 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: loading cluster ID for follower catalog manager: success
I20260924 10:39:30.208137 25744 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: acquiring CA information for follower catalog manager: success
I20260924 10:39:30.209406 25744 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e138dc73f90142bfa7f1fa0d3cf0d50a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:39:30.393841 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:30.389751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.2.193.189:46073)
0924 10:39:30.390060 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:30.390064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:30.390103 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:39:30.390165 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:30.390167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:30.390320 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:30.390415 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:30.390671 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.391195 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.392099 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.392330 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.392412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:30.392702 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:30.393322 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:30.393417 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:30.393563 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 10:39:30.393629 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260924 10:39:30.400453 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:30.397004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.2.193.188:46263)
0924 10:39:30.397327 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:30.397330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:30.397343 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:30.397420 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:30.397422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:30.397538 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:30.397634 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:30.397832 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.398328 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.399103 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.399316 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.399389 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:30.399637 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:30.400065 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:30.400155 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:30.400209 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 10:39:30.400253 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260924 10:39:30.408205 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:30.403851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.2.193.189:46073)
0924 10:39:30.403992 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:30.403997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:30.404310 (+ 313us) server_negotiation.cc:407] Connection header received
0924 10:39:30.404436 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:30.404439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:30.404488 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:30.404580 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:30.404810 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.405628 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.407116 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.407356 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.407491 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:30.407739 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:30.407821 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:30.407891 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:30.408002 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 10:39:30.408064 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:39:30.408303 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:30.404281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.2.193.188:46263)
0924 10:39:30.404391 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:30.404395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:30.404595 (+ 200us) server_negotiation.cc:407] Connection header received
0924 10:39:30.404746 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:30.404749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:30.404808 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:30.404873 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:30.405233 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.405922 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.407090 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.407390 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.407503 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:30.407847 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:30.407923 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:30.407983 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:30.408127 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 10:39:30.408174 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:39:30.408392 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:30.403805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.2.193.190:42737)
0924 10:39:30.404054 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:30.404058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:30.404365 (+ 307us) server_negotiation.cc:407] Connection header received
0924 10:39:30.404478 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:30.404481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:30.404683 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:30.404803 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:30.405026 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.405627 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.406485 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:30.406728 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:30.406836 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:30.407222 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:30.407634 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:30.407721 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:30.408148 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 10:39:30.408221 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260924 10:39:30.411512 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 25475
I20260924 10:39:30.417610 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 25550
I20260924 10:39:30.423404 2822 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBKECxY/build/debug/bin/kudu with pid 25624
2026-09-24T10:39:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3093 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6227 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284521 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST